Virtual reticle for augmented reality systems

ABSTRACT

Systems and methods for displaying a virtual reticle in an augmented or virtual reality environment by a wearable device are described. The environment can include real or virtual objects that may be interacted with by the user through a variety of poses, such as, e.g., head pose, eye pose or gaze, or body pose. The user may select objects by pointing the virtual reticle toward a target object by changing pose or gaze. The wearable device can recognize that an orientation of a user&#39;s head or eyes is outside of a range of acceptable or comfortable head or eye poses and accelerate the movement of the reticle away from a default position and toward a position in the direction of the user&#39;s head or eye movement, which can reduce the amount of movement by the user to align the reticle and target.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/063,077, filed Oct. 5, 2020, entitled “VIRTUAL RETICLE FOR AUGMENTEDREALITY SYSTEMS,” which is a continuation of U.S. application Ser. No.16/170,407, filed Oct. 25, 2018, entitled “VIRTUAL RETICLE FOR AUGMENTEDREALITY SYSTEMS,” which claims priority to U.S. Provisional PatentApplication No. 62/578,094, filed Oct. 27, 2017, entitled “VIRTUALRETICLE FOR AUGMENTED REALITY SYSTEMS,” each of which is herebyincorporated by reference herein in its entirety.

COPYRIGHT STATEMENT

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD

The present disclosure relates to virtual reality and augmented realityimaging and visualization systems and more particularly to displaying avirtual reticle based on head pose.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality”, “augmentedreality”, or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Various embodiments of systems and methods for displaying a virtualreticle are disclosed. An environment can include real or virtualobjects that may be interacted with by a user using a virtual reticle.The position or orientation of the user's head, eyes, shoulders, chest,arms or other body parts can dictate the position or speed-of-movementof the virtual reticle within the environment, and the user may selector point to an object by directing the virtual reticle toward orfocusing the reticle on a target object. The wearable device canrecognize that the user's head, eye(s), shoulders, chest, arm(s) or thelike are positioned or orientated uncomfortably or otherwiseundesirably. Responsive to the recognition of the uncomfortable orundesirable pose, the system can adjust a position or speed-of-movementof the virtual reticle to aid in desirably positioning the virtualreticle, thereby decreasing a likelihood that the user's pose remainsuncomfortable or undesirable.

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description purports to defineor limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes.

FIG. 4 schematically illustrates an example of a waveguide stack foroutputting image information to a user.

FIG. 5 shows example exit beams that may be outputted by a waveguide.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield.

FIG. 7 is a block diagram of an example of a wearable system.

FIG. 8 is a process flow diagram of an example of a method of renderingvirtual content in relation to recognized objects.

FIG. 9 is a block diagram of another example of a wearable system.

FIG. 10 is a process flow diagram of an example of a method fordetermining user input to a wearable system.

FIG. 11 is a process flow diagram of an example of a method forinteracting with a virtual user interface.

FIG. 12A schematically illustrates an example of a field of regard(FOR), a field of view (FOV) of a world camera, a field of view of auser, and a field of fixation of a user.

FIG. 12B schematically illustrates an example of virtual objects in auser's field of view and virtual objects in a field of regard.

FIGS. 13A and 13B demonstrate examples of accelerating movement of areticle responsive to changes in head pose vertically (FIG. 13A) andhorizontally (FIG. 13B).

FIG. 14 illustrates examples of adjusting position of a reticle based onthe user's head pose.

FIG. 15 demonstrates an example relationship between a user's head poseand an adjustment to position of a reticle.

FIG. 16 illustrates a flowchart for an example reticle positionadjustment process.

FIG. 17 illustrates a flowchart for an example reticle adjustmentprocess.

FIG. 18 illustrates a flowchart for another example reticle adjustmentprocess.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure. Additionally, the figures in thepresent disclosure are for illustration purposes and are not to scale.

DETAILED DESCRIPTION Overview

A wearable device can include a display for presenting an interactiveVR/AR/MR environment. The VR/AR/MR environment can include data elementsthat may be interacted with by the user through a variety of poses, suchas, e.g., head pose, eye gaze, or body pose. The VR/AR/MR environmentmay include a reticle that corresponds to the user's current positionwith respect to the user's field of view (e.g., the extent of theVR/AR/MR environment that is seen at any given moment). For example, thereticle may represent the user's direction of gaze. When the user movesaround (e.g., by moving eyes, head, body, or any combination thereof),the reticle may also move with the user. The reticle may point at one ormore objects, and the user may select a target object to which thereticle is pointing. For example, the user may move his or her head topoint the reticle to an object to be selected and then click a hand-helduser input totem to select the object. The wearable display device canperform an appropriate action on the selected object (e.g., move theobject, highlight or enlarge the object, call up information about theobject, display a menu of actions that can be performed that areassociated with the object, etc.).

At times, the user may desire to reorient his or her head (non-limitingexample: to select an object that is high in the environment (e.g., theceiling), low to the ground (e.g., the floor or the user's feet), far tothe right, far to the left, etc.), which may require the user to bend,twist or crane his or her neck such that the reticle is positioned atthe desired object. The bending, twisting, and/or craning of the user'sneck can result in, among other things, neck strain or discomfort duringuse of the wearable device. Accordingly, the wearable system canrecognize that an orientation of a user's head is outside of a range ofacceptable (e.g., comfortable, non-straining, etc.) head poses. As aresult, to assist the user in moving or adjusting the reticle, thewearable system may modify or accelerate the movement (e.g., modify anangle) of the reticle in a direction corresponding to a direction towhich the user's head is moving. By modifying or accelerating themovement of the reticle, the wearable system advantageously reduces adegree to which the user must bend, twist or crane his or her neck toalign the reticle and target object, thereby reducing a likelihood ofneck strain or discomfort.

As an example, if the user looks upward toward an object near theceiling, the wearable display device may move the reticle from a defaultreticle position (non-limiting example: near the center of the field ofview (FOV) of the display) to another position, such as toward the topof the user's FOV. Likewise, if the user looks downward, the wearabledisplay device may move the reticle from a default position in the FOVto another position, such as near the bottom of the user's FOV. Thewearable display device may similarly reposition the reticle if the userlooks rightward or leftward (e.g., moving the reticle to the right sideor left side, respectively, of the FOV), diagonally up to the right orleft, diagonally down to the right or left, etc. The default reticleposition is not limited to the center of the FOV. Rather, any locationwithin the FOV can be utilized as the default reticle position.

The user may desire to use his or her eyes to adjust the reticle. Thewearable device can collect eye data such as eye images (e.g., via aneye camera in an inward-facing imaging system of the wearable device).The wearable system can calculate the user's eye gaze direction based ona mapping matrix that provides an association between the user's eyegaze and a gaze vector (which can indicate the user's direction ofgaze). As the user's eyes moves, the wearable system can determine theuser's eye gaze direction and the user's field of view or the reticlemay move in response to changes in the eye gaze direction. If the userdesires to view or target an object that is high, low, far left, farright, and so on using changes in eye gaze rather than or in addition tochanges to head pose, the movement of the user's eyes and/or thefixation of the eyes at a particular location can result in, among otherthings, eye strain, discomfort, or headaches. Accordingly, the wearablesystem can recognize that an orientation of a user's eye is outside of arange of acceptable (e.g., comfortable, non-straining, etc.) eye poses.As a result, to assist the user in moving or adjusting the reticle, thewearable system may accelerate the movement (e.g., modify an angle) ofthe reticle in a direction corresponding to a direction to which theuser's eyes are moving. By accelerating the movement of the reticle, thewearable system advantageously reduces a likelihood of eyestrain,headaches, eye discomfort and the like.

By providing a process in which the user can align the virtual reticlewith a target using a combination of head pose or eye gaze, the wearablesystem can provide an intuitive process by which the user can morereadily make the alignment while reducing or minimizing neck strain oreye strain.

Examples of 3D Display of a Wearable System

A wearable system (also referred to herein as an augmented reality (AR)system) can be configured to present two-dimensional (2D) orthree-dimensional (3D) virtual images to a user. The images may be stillimages, frames of a video, or a video, in combination or the like. Thewearable system can include a wearable device that can present a VR, AR,or MR environment, alone or in combination, for user interaction. Thewearable device can be a head-mounted device (HMD) which is usedinterchangeably as an AR device (ARD).

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1, an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it may bedesirable for each point in the display's visual field to generate anaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane or based on observing differentimage features on different depth planes being out of focus. Asdiscussed elsewhere herein, such depth cues provide credible perceptionsof depth.

FIG. 2 illustrates an example of wearable system 200. The wearablesystem 200 includes a display 220, and various mechanical and electronicmodules and systems to support the functioning of display 220. Thedisplay 220 may be coupled to a frame 230, which is wearable by a user,wearer, or viewer 210. The display 220 can be positioned in front of theeyes of the user 210. The display 220 can present AR/VR/MR content to auser. The display 220 can comprise a head mounted display (HMD) that isworn on the head of the user. In some embodiments, a speaker 240 iscoupled to the frame 230 and positioned adjacent the ear canal of theuser (in some embodiments, another speaker, not shown, is positionedadjacent the other ear canal of the user to provide for stereo/shapeablesound control).

The wearable system 200 can include an outward-facing imaging system 464(shown in FIG. 4) which observes the world in the environment around theuser. The wearable system 200 can also include an inward-facing imagingsystem 462 (shown in FIG. 4) which can track the eye movements of theuser. The inward-facing imaging system may track either one eye'smovements or both eyes' movements. The inward-facing imaging system 462may be attached to the frame 230 and may be in electrical communicationwith the processing modules 260 or 270, which may process imageinformation acquired by the inward-facing imaging system to determine,e.g., the pupil diameters or orientations of the eyes, eye movements oreye pose of the user 210.

As an example, the wearable system 200 can use the outward-facingimaging system 464 or the inward-facing imaging system 462 to acquireimages of a pose of the user. The images may be still images, frames ofa video, or a video, in combination or the like.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system or theoutward-facing imaging system), microphones, inertial measurement units(IMUs) (e.g., accelerometers, gravitometers, magnetometers, etc.),compasses, global positioning system (GPS) units, radio devices, orgyroscopes; or b) acquired or processed using remote processing module270 or remote data repository 280, possibly for passage to the display220 after such processing or retrieval. The local processing and datamodule 260 may be operatively coupled by communication links 262 or 264,such as via wired or wireless communication links, to the remoteprocessing module 270 or remote data repository 280 such that theseremote modules are available as resources to the local processing anddata module 260. In addition, remote processing module 280 and remotedata repository 280 may be operatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data and/or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

The human visual system is complicated and providing a realisticperception of depth is challenging. Without being limited by theory, itis believed that viewers of an object may perceive the object as beingthree-dimensional due to a combination of vergence and accommodation.Vergence movements (i.e., rolling movements of the pupils toward or awayfrom each other to converge the lines of sight of the eyes to fixateupon an object) of the two eyes relative to each other are closelyassociated with focusing (or “accommodation”) of the lenses of the eyes.Under normal conditions, changing the focus of the lenses of the eyes,or accommodating the eyes, to change focus from one object to anotherobject at a different distance will automatically cause a matchingchange in vergence to the same distance, under a relationship known asthe “accommodation-vergence reflex.” Likewise, a change in vergence willtrigger a matching change in accommodation, under normal conditions.Display systems that provide a better match between accommodation andvergence may form more realistic and comfortable simulations ofthree-dimensional imagery.

FIG. 3 illustrates aspects of an approach for simulating athree-dimensional imagery using multiple depth planes. With reference toFIG. 3, objects at various distances from eyes 302 and 304 on the z-axisare accommodated by the eyes 302 and 304 so that those objects are infocus. The eyes 302 and 304 assume particular accommodated states tobring into focus objects at different distances along the z-axis.Consequently, a particular accommodated state may be said to beassociated with a particular one of depth planes 306, which has anassociated focal distance, such that objects or parts of objects in aparticular depth plane are in focus when the eye is in the accommodatedstate for that depth plane. In some embodiments, three-dimensionalimagery may be simulated by providing different presentations of animage for each of the eyes 302 and 304, and also by providing differentpresentations of the image corresponding to each of the depth planes.While shown as being separate for clarity of illustration, it will beappreciated that the fields of view of the eyes 302 and 304 may overlap,for example, as distance along the z-axis increases. In addition, whileshown as flat for the ease of illustration, it will be appreciated thatthe contours of a depth plane may be curved in physical space, such thatall features in a depth plane are in focus with the eye in a particularaccommodated state. Without being limited by theory, it is believed thatthe human eye typically can interpret a finite number of depth planes toprovide depth perception. Consequently, a highly believable simulationof perceived depth may be achieved by providing, to the eye, differentpresentations of an image corresponding to each of these limited numberof depth planes.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 4400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2,with FIG. 4 schematically showing some parts of that wearable system 200in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2.

With continued reference to FIG. 4, the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light exits an output surfaceof the image injection devices 420, 422, 424, 426, 428 and is injectedinto a corresponding input edge of the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, a single beam of light (e.g., acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 410 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 or 270 (illustrated in FIG. 2) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top or bottom major surfaces, or maybe disposed directly in the volume of the waveguides 440 b, 438 b, 436b, 434 b, 432 b. In some embodiments, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in a layer ofmaterial that is attached to a transparent substrate to form thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some other embodiments,the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be a monolithicpiece of material and the light extracting optical elements 440 a, 438a, 436 a, 434 a, 432 a may be formed on a surface or in the interior ofthat piece of material.

With continued reference to FIG. 4, as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it can reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. Such a configuration provides as manyperceived focal planes as there are available waveguide/lens pairings.Both the light extracting optical elements of the waveguides and thefocusing aspects of the lenses may be static (e.g., not dynamic orelectro-active). In some embodiments, either or both may be dynamicusing electro-active features.

With continued reference to FIG. 4, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE has a relatively low diffraction efficiencyso that only a portion of the light of the beam is deflected away towardthe eye 410 with each intersection of the DOE, while the rest continuesto move through a waveguide via total internal reflection. The lightcarrying the image information can thus be divided into a number ofrelated exit beams that exit the waveguide at a multiplicity oflocations and the result is a fairly uniform pattern of exit emissiontoward the eye 304 for this particular collimated beam bouncing aroundwithin a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”state in which they actively diffract, and “off” state in which they donot significantly diffract. For instance, a switchable DOE may comprisea layer of polymer dispersed liquid crystal, in which microdropletscomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets can be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet can beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes ordepth of field may be varied dynamically based on the pupil sizes ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane that is not discernible because the location of thatplane is beyond the depth of focus of the eye may become discernible andappear more in focus with reduction of pupil size and commensurate withthe increase in depth of field. Likewise, the number of spaced apartdepth planes used to present different images to the viewer may bedecreased with the decreased pupil size. For example, a viewer may notbe able to clearly perceive the details of both a first depth plane anda second depth plane at one pupil size without adjusting theaccommodation of the eye away from one depth plane and to the otherdepth plane. These two depth planes may, however, be sufficiently infocus at the same time to the user at another pupil size withoutchanging accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size or orientation, or upon receiving electrical signalsindicative of particular pupil size or orientation. For example, if theuser's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 may beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between theon and off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)of a world camera and the imaging system 464 is sometimes referred to asan FOV camera. The entire region available for viewing or imaging by aviewer may be referred to as the field of regard (FOR). The FOR mayinclude 4π steradians of solid angle surrounding the wearable system 400because the wearer can move his or her body, head, or eyes to perceivesubstantially any direction in space. In other contexts, the wearer'smovements may be more constricted, and accordingly the wearer's FOR maysubtend a smaller solid angle. Images obtained from the outward-facingimaging system 464 can be used to track gestures made by the user (e.g.,hand or finger gestures), detect objects in the world 470 in front ofthe user, and so forth.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter or orientation of only a single eye 410(e.g., using only a single camera per pair of eyes) is determined andassumed to be similar for both eyes of the user. The images obtained bythe inward-facing imaging system 466 may be analyzed to determine theuser's eye pose or mood, which can be used by the wearable system 400 todecide which audio or visual content should be presented to the user.The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using one or more head pose sensors suchas an , (which may comprise an accelerometer, a gyroscope, or amagnetometer), etc.

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.A multi-DOF controller can sense user input in some or all possibletranslations (e.g., left/right, forward/backward, or up/down) orrotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOFcontroller which supports the translation movements may be referred toas a 3DOF while a multi-DOF controller which supports the translationsand rotations may be referred to as 6DOF. In some cases, the user mayuse a finger (e.g., a thumb) to press or swipe on a touch-sensitiveinput device to provide input to the wearable system 400 (e.g., toprovide user input to a user interface provided by the wearable system400). The user input device 466 may be held by the user's hand duringthe use of the wearable system 400. The user input device 466 can be inwired or wireless communication with the wearable system 400.

FIG. 5 shows an example of exit beams outputted by a waveguide. Onewaveguide is illustrated, but it will be appreciated that otherwaveguides in the waveguide assembly 480 may function similarly, wherethe waveguide assembly 480 includes multiple waveguides. Light 520 isinjected into the waveguide 432 b at the input edge 432 c of thewaveguide 432 b and propagates within the waveguide 432 b by TIR. Atpoints where the light 520 impinges on the DOE 432 a, a portion of thelight exits the waveguide as exit beams 510. The exit beams 510 areillustrated as substantially parallel but they may also be redirected topropagate to the eye 410 at an angle (e.g., forming divergent exitbeams), depending on the depth plane associated with the waveguide 432b. It will be appreciated that substantially parallel exit beams may beindicative of a waveguide with light extracting optical elements thatoutcouple light to form images that appear to be set on a depth plane ata large distance (e.g., optical infinity) from the eye 410. Otherwaveguides or other sets of light extracting optical elements may outputan exit beam pattern that is more divergent, which would require the eye410 to accommodate to a closer distance to bring it into focus on theretina and would be interpreted by the brain as light from a distancecloser to the eye 410 than optical infinity.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield. The optical system can include a waveguide apparatus, an opticalcoupler subsystem to optically couple light to or from the waveguideapparatus, and a control subsystem. The optical system can be used togenerate a multi-focal volumetric, image, or light field. The opticalsystem can include one or more primary planar waveguides 632 a (only oneis shown in FIG. 6) and one or more DOEs 632 b associated with each ofat least some of the primary waveguides 632 a. The planar waveguides 632b can be similar to the waveguides 432 b, 434 b, 436 b, 438 b, 440 bdiscussed with reference to FIG. 4. The optical system may employ adistribution waveguide apparatus to relay light along a first axis(vertical or Y-axis in view of FIG. 6), and expand the light's effectiveexit pupil along the first axis (e.g., Y-axis). The distributionwaveguide apparatus may, for example, include a distribution planarwaveguide 622 b and at least one DOE 622 a (illustrated by doubledash-dot line) associated with the distribution planar waveguide 622 b.The distribution planar waveguide 622 b may be similar or identical inat least some respects to the primary planar waveguide 632 b, having adifferent orientation therefrom. Likewise, at least one DOE 622 a may besimilar or identical in at least some respects to the DOE 632 a. Forexample, the distribution planar waveguide 622 b or DOE 622 a may becomprised of the same materials as the primary planar waveguide 632 b orDOE 632 a, respectively. Embodiments of the optical display system 600shown in FIG. 6 can be integrated into the wearable system 200 shown inFIG. 2.

The relayed and exit-pupil expanded light may be optically coupled fromthe distribution waveguide apparatus into the one or more primary planarwaveguides 632 b. The primary planar waveguide 632 b can relay lightalong a second axis, preferably orthogonal to first axis (e.g.,horizontal or X-axis in view of FIG. 6). Notably, the second axis can bea non-orthogonal axis to the first axis. The primary planar waveguide632 b expands the light's effective exit pupil along that second axis(e.g., X-axis). For example, the distribution planar waveguide 622 b canrelay and expand light along the vertical or Y-axis, and pass that lightto the primary planar waveguide 632 b which can relay and expand lightalong the horizontal or X-axis.

The optical system may include one or more sources of colored light(e.g., red, green, and blue laser light) 610 which may be opticallycoupled into a proximal end of a single mode optical fiber 640. A distalend of the optical fiber 640 may be threaded or received through ahollow tube 642 of piezoelectric material. The distal end protrudes fromthe tube 642 as fixed-free flexible cantilever 644. The piezoelectrictube 642 can be associated with four quadrant electrodes (notillustrated). The electrodes may, for example, be plated on the outside,outer surface or outer periphery or diameter of the tube 642. A coreelectrode (not illustrated) may also be located in a core, center, innerperiphery or inner diameter of the tube 642.

Drive electronics 650, for example electrically coupled via wires 660,drive opposing pairs of electrodes to bend the piezoelectric tube 642 intwo axes independently. The protruding distal tip of the optical fiber644 has mechanical modes of resonance. The frequencies of resonance candepend upon a diameter, length, and material properties of the opticalfiber 644. By vibrating the piezoelectric tube 642 near a first mode ofmechanical resonance of the fiber cantilever 644, the fiber cantilever644 can be caused to vibrate, and can sweep through large deflections.

By stimulating resonant vibration in two axes, the tip of the fibercantilever 644 is scanned biaxially in an area filling two-dimensional(2D) scan. By modulating an intensity of light source(s) 610 insynchrony with the scan of the fiber cantilever 644, light emerging fromthe fiber cantilever 644 can form an image. Descriptions of such a setup are provided in U.S. Patent Publication No. 2014/0003762, which isincorporated by reference herein in its entirety.

A component of an optical coupler subsystem can collimate the lightemerging from the scanning fiber cantilever 644. The collimated lightcan be reflected by mirrored surface 648 into the narrow distributionplanar waveguide 622 b which contains the at least one diffractiveoptical element (DOE) 622 a. The collimated light can propagatevertically (relative to the view of FIG. 6) along the distributionplanar waveguide 622 b by TIR, and in doing so repeatedly intersectswith the DOE 622 a. The DOE 622 a preferably has a low diffractionefficiency. This can cause a fraction (e.g., 10%) of the light to bediffracted toward an edge of the larger primary planar waveguide 632 bat each point of intersection with the DOE 622 a, and a fraction of thelight to continue on its original trajectory down the length of thedistribution planar waveguide 622 b via TIR.

At each point of intersection with the DOE 622 a, additional light canbe diffracted toward the entrance of the primary waveguide 632 b. Bydividing the incoming light into multiple outcoupled sets, the exitpupil of the light can be expanded vertically by the DOE 4 in thedistribution planar waveguide 622 b. This vertically expanded lightcoupled out of distribution planar waveguide 622 b can enter the edge ofthe primary planar waveguide 632 b.

Light entering primary waveguide 632 b can propagate horizontally(relative to the view of FIG. 6) along the primary waveguide 632 b viaTIR. As the light intersects with DOE 632 a at multiple points as itpropagates horizontally along at least a portion of the length of theprimary waveguide 632 b via TIR. The DOE 632 a may advantageously bedesigned or configured to have a phase profile that is a summation of alinear diffraction pattern and a radially symmetric diffractive pattern,to produce both deflection and focusing of the light. The DOE 632 a mayadvantageously have a low diffraction efficiency (e.g., 10%), so thatonly a portion of the light of the beam is deflected toward the eye ofthe view with each intersection of the DOE 632 a while the rest of thelight continues to propagate through the primary waveguide 632 b viaTIR.

At each point of intersection between the propagating light and the DOE632 a, a fraction of the light is diffracted toward the adjacent face ofthe primary waveguide 632 b allowing the light to escape the TIR, andemerge from the face of the primary waveguide 632 b. In someembodiments, the radially symmetric diffraction pattern of the DOE 632 aadditionally imparts a focus level to the diffracted light, both shapingthe light wavefront (e.g., imparting a curvature) of the individual beamas well as steering the beam at an angle that matches the designed focuslevel.

Accordingly, these different pathways can cause the light to be coupledout of the primary planar waveguide 632 b by a multiplicity of DOEs 632a at different angles, focus levels, and/or yielding different fillpatterns at the exit pupil. Different fill patterns at the exit pupilcan be beneficially used to create a light field display with multipledepth planes. Each layer in the waveguide assembly or a set of layers(e.g., 3 layers) in the stack may be employed to generate a respectivecolor (e.g., red, blue, green). Thus, for example, a first set of threeadjacent layers may be employed to respectively produce red, blue andgreen light at a first focal depth. A second set of three adjacentlayers may be employed to respectively produce red, blue and green lightat a second focal depth. Multiple sets may be employed to generate afull 3D or 4D color image light field with various focal depths.

Other Components of the Wearable System

In many implementations, the wearable system may include othercomponents in addition or alternatively to the components of thewearable system described above. The wearable system may, for example,include one or more haptic devices or components. The haptic devices orcomponents may be operable to provide a tactile sensation to a user. Forexample, the haptic devices or components may provide a tactilesensation of pressure or texture when touching virtual content (e.g.,virtual objects, virtual tools, other virtual constructs). The tactilesensation may replicate a feel of a physical object which a virtualobject represents, or may replicate a feel of an imagined object orcharacter (e.g., a dragon) which the virtual content represents. In someimplementations, haptic devices or components may be worn by the user(e.g., a user wearable glove). In some implementations, haptic devicesor components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain implementations, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forexample, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate does not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardor virtual trackpad. The user input device 466 (shown in FIG. 4) may bean embodiment of a totem, which may include a trackpad, a touchpad, atrigger, a joystick, a trackball, a rocker or virtual switch, a mouse, akeyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system or other users.

Examples of haptic devices and totems usable with the wearable devices,HMD, and display systems of the present disclosure are described in U.S.Patent Publication No. 2015/0016777, which is incorporated by referenceherein in its entirety.

Example Wearable Systems, Environments, and Interfaces

A wearable system may employ various mapping related techniques in orderto achieve high depth of field in the rendered light fields. In mappingout the virtual world, it is advantageous to know all the features andpoints in the real world to accurately portray virtual objects inrelation to the real world. To this end, FOV images captured from usersof the wearable system can be added to a world model by including newpictures that convey information about various points and features ofthe real world. For example, the wearable system can collect a set ofmap points (such as 2D points or 3D points) and find new map points torender a more accurate version of the world model. The world model of afirst user can be communicated (e.g., over a network such as a cloudnetwork) to a second user so that the second user can experience theworld surrounding the first user.

FIG. 7 is a block diagram of an example of an MR environment 700. The MRenvironment 700 may be configured to receive input (e.g., visual input702 from the user's wearable system, stationary input 704 such as roomcameras, sensory input 706 from various sensors, gestures, totems, eyetracking, user input from the user input device 466 etc.) from one ormore user wearable systems (e.g., wearable system 200 or display system220) or stationary room systems (e.g., room cameras, etc.). The wearablesystems can use various sensors (e.g., accelerometers, gyroscopes,temperature sensors, movement sensors, depth sensors, GPS sensors,inward-facing imaging system, outward-facing imaging system, etc.) todetermine the location and various other attributes of the environmentof the user. This information may further be supplemented withinformation from stationary cameras in the room that may provide imagesor various cues from a different point of view. The image data acquiredby the cameras (such as the room cameras and/or the cameras of theoutward-facing imaging system) may be reduced to a set of mappingpoints.

One or more object recognizers 708 can crawl through the received data(e.g., the collection of points) and recognize or map points, tagimages, attach semantic information to objects with the help of a mapdatabase 710. The map database 710 may comprise various points collectedover time and their corresponding objects. The various devices and themap database can be connected to each other through a network (e.g.,LAN, WAN, etc.) to access the cloud.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects in anenvironment. For example, the object recognizers can recognize faces,persons, windows, walls, user input devices, televisions, other objectsin the user's environment, etc. One or more object recognizers may bespecialized for object with certain characteristics. For example, theobject recognizer 708 a may be used to recognizer faces, while anotherobject recognizer may be used recognize totems.

The object recognitions may be performed using a variety of computervision techniques. For example, the wearable system can analyze theimages acquired by the outward-facing imaging system 464 (shown in FIG.4) to perform scene reconstruction, event detection, video tracking,object recognition, object pose estimation, learning, indexing, motionestimation, or image restoration, etc. One or more computer visionalgorithms may be used to perform these tasks. Non-limiting examples ofcomputer vision algorithms include: Scale-invariant feature transform(SIFT), speeded up robust features (SURF), oriented FAST and rotatedBRIEF (ORB), binary robust invariant scalable keypoints (BRISK), fastretina keypoint (FREAK), Viola-Jones algorithm, Eigenfaces approach,Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm,visual simultaneous location and mapping (vSLAM) techniques, asequential Bayesian estimator (e.g., Kalman filter, extended Kalmanfilter, etc.), bundle adjustment, Adaptive thresholding (and otherthresholding techniques), Iterative Closest Point (ICP), Semi GlobalMatching (SGM), Semi Global Block Matching (SGBM), Feature PointHistograms, various machine learning algorithms (such as e.g., supportvector machine, k-nearest neighbors algorithm, Naive Bayes, neuralnetwork (including convolutional or deep neural networks), or othersupervised/unsupervised models, etc.), and so forth.

In some embodiments, the object recognitions can be performed by avariety of machine learning algorithms. Once trained, the machinelearning algorithm can be stored by the HMD. Some examples of machinelearning algorithms can include supervised or non-supervised machinelearning algorithms, including regression algorithms (such as, forexample, Ordinary Least Squares Regression), instance-based algorithms(such as, for example, Learning Vector Quantization), decision treealgorithms (such as, for example, classification and regression trees),Bayesian algorithms (such as, for example, Naive Bayes), clusteringalgorithms (such as, for example, k-means clustering), association rulelearning algorithms (such as, for example, a-priori algorithms),artificial neural network algorithms (such as, for example, Perceptron),deep learning algorithms (such as, for example, Deep Boltzmann Machine,or deep neural network), dimensionality reduction algorithms (such as,for example, Principal Component Analysis), ensemble algorithms (suchas, for example, Stacked Generalization), and/or other machine learningalgorithms. In some embodiments, individual models can be customized forindividual data sets. For example, the wearable device can generate orstore a base model. The base model may be used as a starting point togenerate additional models specific to a data type (e.g., a particularuser in the telepresence session), a data set (e.g., a set of additionalimages obtained of the user in the telepresence session), conditionalsituations, or other variations. In some embodiments, the wearable HMDcan be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects andsupplement objects with semantic information to give life to theobjects. For example, if the object recognizer recognizes a set ofpoints to be a door, the system may attach some semantic information(e.g., the door has a hinge and has a 90 degree movement about thehinge). If the object recognizer recognizes a set of points to be amirror, the system may attach semantic information that the mirror has areflective surface that can reflect images of objects in the room. Overtime, the map database grows as the system (which may reside locally ormay be accessible through a wireless network) accumulates more data fromthe world. Once the objects are recognized, the information may betransmitted to one or more wearable systems. For example, the MRenvironment 700 may include information about a scene happening inCalifornia. The environment 700 may be transmitted to one or more usersin New York. Based on data received from an FOV camera and other inputs,the object recognizers and other software components can map the pointscollected from the various images, recognize objects etc., such that thescene may be accurately “passed over” to a second user, who may be in adifferent part of the world. The environment 700 may also use atopological map for localization purposes.

FIG. 8 is a process flow diagram of an example of a method 800 ofrendering virtual content in relation to recognized objects. The method800 describes how a virtual scene may be represented to a user of thewearable system. The user may be geographically remote from the scene.For example, the user may be New York, but may want to view a scene thatis presently going on in California, or may want to go on a walk with afriend who resides in California.

At block 810, the wearable system may receive input from the user andother users regarding the environment of the user. This may be achievedthrough various input devices, and knowledge already possessed in themap database. The user's FOV camera, sensors, GPS, eye tracking, etc.,convey information to the system at block 810. The system may determinesparse points based on this information at block 820. The sparse pointsmay be used in determining pose data (e.g., head pose, eye pose, bodypose, or hand gestures) that can be used in displaying and understandingthe orientation and position of various objects in the user'ssurroundings. The object recognizers 708 a-708 n may crawl through thesecollected points and recognize one or more objects using a map databaseat block 830. This information may then be conveyed to the user'sindividual wearable system at block 840, and the desired virtual scenemay be accordingly displayed to the user at block 850. For example, thedesired virtual scene (e.g., user in CA) may be displayed at theappropriate orientation, position, etc., in relation to the variousobjects and other surroundings of the user in New York.

FIG. 9 is a block diagram of another example of a wearable system. Inthis example, the wearable system 900 comprises a map, which may includemap data for the world. The map may partly reside locally on thewearable system, and may partly reside at networked storage locationsaccessible by wired or wireless network (e.g., in a cloud system). Apose process 910 may be executed on the wearable computing architecture(e.g., processing module 260 or controller 460) and utilize data fromthe map to determine position and orientation of the wearable computinghardware or user. Pose data may be computed from data collectedreal-time as the user is experiencing the system and operating in theworld. The data may comprise images, data from sensors (such as inertialmeasurement units (IMUs), which may comprise an accelerometer, agyroscope, a magnetometer, or combinations of such components) andsurface information pertinent to objects in the real or virtualenvironment.

A sparse point representation may be the output of a simultaneouslocalization and mapping (SLAM or V-SLAM, referring to a configurationwherein the input is images/visual only) process. The system can beconfigured to not only find out where in the world the variouscomponents are, but what the world is made of. Pose may be a buildingblock that achieves many goals, including populating the map and usingthe data from the map.

In one embodiment, a sparse point position may not be completelyadequate on its own, and further information may be needed to produce amultifocal AR, VR, or MR experience. Dense representations, generallyreferring to depth map information, may be utilized to fill this gap atleast in part. Such information may be computed from a process referredto as Stereo 940, wherein depth information is determined using atechnique such as triangulation or time-of-flight sensing. Imageinformation and active patterns (such as infrared patterns created usingactive projectors) may serve as input to the Stereo process 940. Asignificant amount of depth map information may be fused together, andsome of this may be summarized with a surface representation. Forexample, mathematically definable surfaces may be efficient (e.g.,relative to a large point cloud) and digestible inputs to otherprocessing devices like game engines. Thus, the output of the stereoprocess (e.g., a depth map) 940 may be combined in the fusion process930. Pose may be an input to this fusion process 930 as well, and theoutput of fusion 930 becomes an input to populating the map process 920.Sub-surfaces may connect with each other, such as in topographicalmapping, to form larger surfaces, and the map becomes a large hybrid ofpoints and surfaces.

To resolve various aspects in a mixed reality process 960, variousinputs may be utilized. For example, in the embodiment depicted in FIG.9, Game parameters may be inputs to determine that the user of thesystem is playing a monster battling game with one or more monsters atvarious locations, monsters dying or running away under variousconditions (such as if the user shoots the monster), walls or otherobjects at various locations, and the like. The world map may includeinformation regarding where such objects are relative to each other, tobe another valuable input to mixed reality. Pose relative to the worldbecomes an input as well and plays a key role to almost any interactivesystem.

Controls or inputs from the user are another input to the wearablesystem 900. As described herein, user inputs can include visual input,gestures, totems, audio input, sensory input, etc. In order to movearound or play a game, for example, the user may need to instruct thewearable system 900 regarding what he or she wants to do. Beyond justmoving oneself in space, there are various forms of user controls thatmay be utilized. In one embodiment, a totem (e.g. a user input device),or an object such as a toy gun may be held by the user and tracked bythe system. The system preferably will be configured to know that theuser is holding the item and understand what kind of interaction theuser is having with the item (e.g., if the totem or object is a gun, thesystem may be configured to understand location and orientation, as wellas whether the user is clicking a trigger or other sensed button orelement which may be equipped with a sensor, such as an IMU, which mayassist in determining what is going on, even when such activity is notwithin the field of view of any of the cameras.)

Hand gesture tracking or recognition may also provide input information.The wearable system 900 may be configured to track and interpret handgestures for button presses, for gesturing left or right, stop, grab,hold, etc. For example, in one configuration, the user may want to flipthrough emails or a calendar in a non-gaming environment, or do a “fistbump” with another person or player. The wearable system 900 may beconfigured to leverage a minimum amount of hand gesture, which may ormay not be dynamic. For example, the gestures may be simple staticgestures like open hand for stop, thumbs up for ok, thumbs down for notok; or a hand flip right, or left, or up/down for directional commands.

Eye tracking is another input (e.g., tracking where the user is lookingto control the display technology to render at a specific depth orrange). In one embodiment, vergence of the eyes may be determined usingtriangulation, and then using a vergence/accommodation model developedfor that particular person, accommodation may be determined.

With regard to the camera systems, the example wearable system 900 shownin FIG. 9 can include three pairs of cameras: a relative wide FOV orpassive SLAM pair of cameras arranged to the sides of the user's face, adifferent pair of cameras oriented in front of the user to handle thestereo imaging process 940 and also to capture hand gestures andtotem/object tracking in front of the user's face. The FOV cameras andthe pair of cameras for the stereo process 940 may be a part of theoutward-facing imaging system 464 (shown in FIG. 4). The wearable system900 can include eye tracking cameras (which may be a part of aninward-facing imaging system 462 shown in FIG. 4) oriented toward theeyes of the user in order to triangulate eye vectors and otherinformation. The wearable system 900 may also comprise one or moretextured light projectors (such as infrared (IR) projectors) to injecttexture into a scene.

FIG. 10 is a process flow diagram of an example of a method 1000 fordetermining user input to a wearable system. In this example, the usermay interact with a totem. The user may have multiple totems. Forexample, the user may have designated one totem for a social mediaapplication, another totem for playing games, etc. At block 1010, thewearable system may detect a motion of a totem. The movement of thetotem may be recognized through the outward facing system or may bedetected through sensors (e.g., haptic glove, image sensors, handtracking devices, eye-tracking cameras, head pose sensors, etc.).

Based at least partly on the detected gesture, eye pose, head pose, orinput through the totem, the wearable system detects a position,orientation, and/or movement of the totem (or the user's eyes or head orgestures) with respect to a reference frame, at block 1020. Thereference frame may be a set of map points based on which the wearablesystem translates the movement of the totem (or the user) to an actionor command. At block 1030, the user's interaction with the totem ismapped. Based on the mapping of the user interaction with respect to thereference frame 1020, the system determines the user input at block1040.

For example, the user may move a totem or physical object back and forthto signify turning a virtual page and moving on to a next page or movingfrom one user interface (UI) display screen to another UI screen. Asanother example, the user may move their head or eyes to look atdifferent real or virtual objects in the user's FOR. If the user's gazeat a particular real or virtual object is longer than a threshold time,the real or virtual object may be selected as the user input. In someimplementations, the vergence of the user's eyes can be tracked and anaccommodation/vergence model can be used to determine the accommodationstate of the user's eyes, which provides information on a depth plane onwhich the user is focusing. In some implementations, the wearable systemcan use ray-casting techniques to determine which real or virtualobjects are along the direction of the user's head pose or eye pose. Invarious implementations, the ray casting techniques can include castingthin, pencil rays with substantially little transverse width or castingrays with substantial transverse width (e.g., cones or frustums).

The user interface may be projected by the display system as describedherein (such as the display 220 in FIG. 2). It may also be displayedusing a variety of other techniques such as one or more projectors. Theprojectors may project images onto a physical object such as a canvas ora globe. Interactions with user interface may be tracked using one ormore cameras external to the system or part of the system (such as,e.g., using the inward-facing imaging system 462 or the outward-facingimaging system 464).

FIG. 11 is a process flow diagram of an example of a method 1100 forinteracting with a virtual user interface. The method 1100 may beperformed by the wearable system described herein.

At block 1110, the wearable system may identify a particular UI. Thetype of UI may be predetermined by the user. The wearable system mayidentify that a particular UI needs to be populated based on a userinput (e.g., gesture, visual data, audio data, sensory data, directcommand, etc.). At block 1120, the wearable system may generate data forthe virtual UI. For example, data associated with the confines, generalstructure, shape of the UI etc., may be generated. In addition, thewearable system may determine map coordinates of the user's physicallocation so that the wearable system can display the UI in relation tothe user's physical location. For example, if the UI is body centric,the wearable system may determine the coordinates of the user's physicalstance, head pose, or eye pose such that a ring UI can be displayedaround the user or a planar UI can be displayed on a wall or in front ofthe user. If the UI is hand centric, the map coordinates of the user'shands may be determined. These map points may be derived through datareceived through the FOV cameras, sensory input, or any other type ofcollected data.

At block 1130, the wearable system may send the data to the display fromthe cloud or the data may be sent from a local database to the displaycomponents. At block 1140, the UI is displayed to the user based on thesent data. For example, a light field display can project the virtual UIinto one or both of the user's eyes. Once the virtual UI has beencreated, the wearable system may simply wait for a command from the userto generate more virtual content on the virtual UI at block 1150. Forexample, the UI may be a body centric ring around the user's body. Thewearable system may then wait for the command (a gesture, a head or eyemovement, input from a user input device, etc.), and if it is recognized(block 1160), virtual content associated with the command may bedisplayed to the user (block 1170). As an example, the wearable systemmay wait for user's hand gestures before mixing multiple steam tracks.

Additional examples of wearable systems, UIs, and user experiences (UX)are described in U.S. Patent Publication No. 2015/0016777, which isincorporated by reference herein in its entirety.

Example Objects in the Field of Regard (FOR) and Field of View (FOV)

FIG. 12A schematically illustrates an example of a field of regard (FOR)1200, a field of view (FOV) of a world camera 1270, a field of view of auser 1250, and a field of fixation of a user 1290. As described withreference to FIG. 4, the FOR 1200 comprises a portion of the environmentaround the user that is capable of being perceived by the user via thewearable system. The FOR may include 4π steradians of solid anglesurrounding the wearable system 400 because the wearer can move hisbody, head, or eyes to perceive substantially any direction in space. Inother contexts, the wearer's movements may be more constricted, andaccordingly the wearer's FOR may subtend a smaller solid angle.

The field of view of a world camera 1270 can include a portion of theuser's FOR that is currently observed by an outward-facing imagingsystem 464. With reference to FIG. 4, the field of view of a worldcamera 1270 may include the world 470 that is observed by the wearablesystem 400 at a given time. The size of the FOV of the world camera 1270may depend on the optical characteristics of the outward-facing imagingsystem 464. For example, the outward-facing imaging system 464 caninclude a wide angle camera that can image a 190 degree space around theuser. In certain implementations, the FOV of the world camera 1270 maybe larger than or equal to a natural FOV of a user's eyes.

The FOV of a user 1250 can include the portion of the FOR 1200 that auser perceives at a given time. The FOV can depend on the size oroptical characteristics of the display of a wearable device. Forexample, an AR display may include optics that only provides ARfunctionality when the user looks through a particular portion of thedisplay. The FOV 1250 may correspond to the solid angle that isperceivable by the user when looking through an AR display such as,e.g., the stacked waveguide assembly 480 (FIG. 4) or the planarwaveguide 600 (FIG. 6). In certain embodiments, the FOV of a user 1250may be smaller than the natural FOV of the user's eyes.

The wearable system can also determine a user's field of fixation (FOF)1290. The FOF 1290 can include a portion of the FOV 1250 at which theuser's eyes can fixate (e.g., maintain visual gaze at that portion). TheFOF 1290 can be smaller than the FOV 1250 of a user, for example, theFOF may be a few degrees to about 5 degrees across. As a result, theuser can perceive some virtual objects in the FOV 1250 that are not inthe field of fixation 1290 but which are in a peripheral FOV of theuser.

FIG. 12B schematically illustrates an example of virtual objects in auser's FOV 1250 and virtual objects in a FOR 1200. In FIG. 12B, the FOR1200 can include a group of objects (e.g. 1210, 1220, 1230, 1242, and1244) which can be perceived by the user via the wearable system. Theobjects within the FOR 1200 may be virtual and/or physical objects. Forexample, the FOR 1200 may include physical object such as a chair, asofa, a wall, etc. The virtual objects may include operating systemobjects such as e.g., a recycle bin for deleted files, a terminal forinputting commands, a file manager for accessing files or directories,an icon, a menu, an application for audio or video streaming, anotification from an operating system, and so on. The virtual objectsmay also include objects in an application such as e.g., avatars,virtual objects in games, graphics or images, etc. Some virtual objectscan be both an operating system object and an object in an application.In some embodiments, the wearable system can add virtual elements to theexisting physical objects. For example, the wearable system may add avirtual menu associated with a television in the room, where the virtualmenu may give the user the option to turn on or change the channels ofthe television using the wearable system.

A virtual object may be a three-dimensional (3D), two-dimensional (2D),or one-dimensional (1D) object. For example, the virtual object may be a3D coffee mug (which may represent a virtual control for a physicalcoffee maker). The virtual object may also be a 2D graphicalrepresentation of a clock (displaying current time to the user). In someimplementations, one or more virtual objects may be displayed within (orassociated with) another virtual object. A virtual coffee mug may beshown inside of a user interface plane, although the virtual coffee mugappears to be 3D within this 2D planar virtual space.

The objects in the user's FOR can be part of a world map as describedwith reference to FIG. 9. Data associated with objects (e.g. location,semantic information, properties, etc.) can be stored in a variety ofdata structures such as, e.g., arrays, lists, trees, hashes, graphs, andso on. The index of each stored object, wherein applicable, may bedetermined, for example, by the location of the object. For example, thedata structure may index the objects by a single coordinate such as theobject's distance from a fiducial position (e.g., how far to the left orright of the fiducial position, how far from the top or bottom of thefiducial position, or how far depth-wise from the fiducial position).The fiducial position may be determined based on the user's position(such as the position of the user's head or eyes). The fiducial positionmay also be determined based on the position of a virtual or physicalobject (such as a target object) in the user's environment. That way,the 3D space in the user's environment may be collapsed into a 2D userinterface where the virtual objects are arranged in accordance with theobject's distance from the fiducial position.

Utilization of a Reticle

With continued reference to FIG. 12B, the VR/AR/MR system can display avirtual reticle 1256 which may include a movable indicator identifying aposition of a user within the FOV 1250. For example, the reticle 1256may represent a direction of gaze of a user such as a field of fixation,a point that will be affected by input from the user, or the like. Theappearance of a reticle 1256 can take on any of a variety of differentcolors, outlines, shapes, symbols, sizes, images, graphics, incombination or the like. For example, the reticle 1256 may take avariety of shapes such as a cursor, a geometric cone, a narrow beam , anarrow, an oval, a circle, a bullseye, a polygon, or other 1D, 2D, or 3Dshapes. The reticle 1256 may be a virtual object that is fixed within arig space (e.g., a coordinate system associated with the wearable devicesuch as a Cartesian x-y-z coordinate system), but also may be capable ofbeing fixed within the user's 3D environment. The reticle 1256 may berepresented by a virtual object that the user can drag and drop (e.g.,from a position in a rig space) to a specific position within the user's3D space.

A user can move his or her body, head, or eyes to move the reticle 1256.For example, a change in the user's pose (e.g., head pose, body pose, oreye gaze) may alter the location of the reticle 1256 within FOV 1250and/or alter what is shown or observable in the FOV 1250. Similarly, thereticle 1256 may be controlled though a user input device such as atrackpad, a touchscreen, a joystick, a multiple degree-of-freedom (DOF)controller, a capacitive sensing device, a game controller, a keyboard,a mouse, a directional pad (D-pad), a wand, a haptic device, a totem(e.g., functioning as a virtual user input device), and so forth. Forexample, as the user moves his hand on a user input device, the reticle1256 may move from a first position to a second position.

The reticle 1256 may be used to select, view, or point to an object,such as one of objects 1210, 1242, 1244, 1230, 1220, by moving thereticle 1256 such that it hovers over or otherwise points to the targetobject. For example, to effectively align a target object and thereticle 1256, the user may tilt, turn or otherwise reorient his or herhead to a pose corresponding to the location of the target object. Oncethe reticle 1256 and the target object are aligned, the user may selectthe target object to which the reticle 1256 is hovering or pointing. Incertain embodiments, the target object may also receive a focusindicator (e.g., virtual rays emanating from the reticle 1256 orselected object or other graphical highlighting).

Accelerating a Position of a Reticle

FIGS. 13A-13B demonstrate examples of accelerating movement of a reticleresponsive to changes in head pose. The environments 1300A, 1300Binclude a user 1304 wearing wearable system 1320, such as system 200 ofFIG. 2, and further include a representation of the wearable system's1320 field of regard (FOR) 1310. As described with reference to FIG. 4,the FOR 1310 comprises a portion of the environment around the user 1304that is capable of being perceived by the user via the wearable system1320.

The user 1304 can move his or her body, head, or eyes to perceivesubstantially any direction in the FOR 1310. For example, the user'shead may have multiple degrees of freedom. As the user's head moves(e.g., tilts or turns), the user's head pose changes. Changes in headpose can be quantified by determining changes in an angle with respectto a reference head pose vector. The reference head pose can be any headpose of the user. For example, when considering coordinate values in thex-y-z coordinate systems shown in FIGS. 13A and 13B, a reference headpose vector may correspond to a position of the user's head when each ofthe coronal plane of the user's head (e.g., vertical plane that dividesthe body into belly and back sections), the sagittal plane of the user'shead (e.g., an anatomical plane which divides the body into right andleft parts), and the axial plane of the user's head (e.g., a plane thatdivides the body into superior and inferior parts, roughly perpendicularto spine) of the user's head are orthogonal to one another.

In some applications, a user may look to a particular direction morefrequently, and the system may adjust the reference head pose to reduceor ameliorate the neck strain. For example, a game relating to shootingflying birds and a star gazing application that teaches users aboutconstellations may both involve a user periodically or constantlylooking upward. Similarly, a game of chess may involve a userperiodically or constantly looking downward. Other games or applicationsmay involve a user periodically or constantly looking left, right,diagonally, or the like. Accordingly, in some cases, the reference headpose can be updated to hedge toward a head pose associated with thismore frequent direction. For example, the reference head pose can be anaverage or most-common head pose of a user over a particular period oftime or an average or most-common head pose of a user while the useruses the particular application or plays the particular game. Byconfiguring the reference head pose to hedge toward or match a head poseassociated with a common or average head pose of the user, the systemcan reduce or ameliorate the neck strain.

The coordinate systems in FIGS. 13A and 13B show three angular degreesof freedom (e.g., yaw, pitch, and roll) that can be used for measuringthe head pose relative to the reference head pose vector. For example,the user's head can tilt forward or backward (e.g., pitching), turn leftor right (e.g., yawing), or tilt side to side (e.g., rolling), and anangular difference between the new head pose and the reference head posevector can be determined to quantify the changes. In otherimplementations, other techniques or angular representations formeasuring head pose can be used, for example, quaternion angles or anytype of Euler angle system.

As the user's pose changes (e.g., body, head or eye pose), the user'sFOV may correspondingly change, and any objects within the FOV may alsochange. FIG. 13A illustrates three FOVs 1350 a, 1350 b, 1350 c which areschematically represented by rectangles, and which can correspond to aportion of the user's FOR 1310 that is observed at a distinct period intime. In each of the three illustrated scenes 1350 a, 1350 b, 1350 c, auser can perceive a reticle 1356 via the display 1320. In addition, asillustrated in FOVs 1350 b, 1350 c, a user can perceive target objects,such as cylinder 1382 or box 1380, which can represent a virtual or aphysical object that is at a given location in the user's environment.As the user adjusts his or her head (e.g., pitching, yawing or rolling),the system may adjust the user's FOV such that the user perceives thathe or she is tilting or turning his or her head in the VR/AR/MRenvironment.

As a non-limiting example, the user 1304 may desire to view or interactwith object 1380 and/or object 1382. For simplicity, an initial pose ofthe user's head 1348 corresponds to the reference head pose vector, asdescribed herein. In addition, at the initial head pose 1348, the userperceives FOV 1350 a. Because no objects are perceivable within FOV 1350a, the user may begin to scan the FOR 1310 to search for the objects1380, 1382. As the user tilts his or her head down toward the floor, theuser's FOV correspondingly adjusts such that the user may eventuallyperceive object 1382 within FOV 1350 b. The user may continue to scanthe FOR 1310 to search for more objects, for instance, by tilting his orher head up toward the ceiling. At some later point in time, the usermay adjust his or her head pose such that the user perceives FOV 1350 c.As illustrated, the object 1380 is perceivable in FOV 1350 c. In someinstances, in addition to simply perceiving the object, the user mayalso select the object using the reticle 1356.

The reticle 1356 can be positioned on a target object using variousmethods. As a first example, the user's FOV may be temporarily orpermanently fixed within the FOR. Accordingly, a change in the user'spose (e.g., head pose, body pose, eye pose) may cause a reticle to movewithin the user's FOV (e.g., relative to a default reticle position,which may be near the center of the FOV), but does not cause the user'sFOV to change. For example, if the user looks up or tilts his or herhead back, the movement may cause the reticle to move up within the FOV,such as moving closer to the top of the FOV. Accordingly, in examplessuch as these, to select an object using the reticle, the object mayneed to be within the FOV and the user may need to position his or herhead such that the reticle points to the target object.

As a second example, the position of a reticle within a FOV may betemporarily or permanently fixed. Accordingly, a change in the user'spose may alter the user's FOV, but does not change the location of thereticle within the FOV. For example, a reticle may be fixed or locked ata location (e.g., the center) of the user's FOV, such as illustrated inFOV 1350 a. As the user's pose changes, the FOV may change, but thereticle remains at a default position (e.g., the center) of the dynamicFOV. Accordingly, to select an object using the reticle, the user mayneed to position his or her head such that the target object is atcenter of the FOV (e.g., the position of the reticle).

As a third example, the reticle can be positioned on a target objectusing a combination of the first two examples. The reticle may betemporarily fixed at a position within the FOV. For example, the reticlemay be fixed when the user's head pose is between a minimum head posethreshold and a maximum head pose threshold (e.g., satisfies a minimumhead pose threshold and does not satisfy a maximum head pose threshold).As the user's pose changes, the user's FOV may change while the positionof the reticle within the FOV remains the same. However, as the user'shead moves toward an uncomfortable or otherwise undesired pose (e.g.,the user's head pose does not satisfy a minimum head pose threshold orsatisfies a maximum head pose threshold), the reticle may become unfixedand may be free to move within the user's FOV. For example, the wearablesystem may accelerate the movement of the reticle in a directioncorresponding to a direction to which the user's head is moving. In someinstances, this acceleration may reduce a likelihood of neck strain ordiscomfort, because the reticle moves toward the position the user ismoving his or her head, thereby reducing or minimizing the amount ofhead movement needed to position the reticle with a target object.Although the reticle is accelerated, in some embodiments, the reticle isnot accelerated past a threshold position (e.g., a position within theFOV). For example, in some cases, the reticle is not accelerated out ofthe user's FOV. This advantageously aids in reducing a likelihood ofneck strain or discomfort while also retaining the user's ability tointeract with the objects in the FOV via the reticle.

At times when the user desires to reorient his or her head (non-limitingexample: to select an object that is high in the air (e.g., theceiling), low to the ground (e.g., the floor), far to the right, far tothe left, etc.), the user may have to bend, twist or crane his or herneck such that the reticle 1356 is positioned at the desired object. Thebending, twisting, and/or craning of the user's neck can result in,among other things, neck strain or discomfort. Accordingly, the wearablesystem can recognize that an orientation of a user's head is outside ofa range (e.g., below a minimum head pose threshold, above a maximum headpose threshold) of acceptable (e.g., comfortable, non-straining, etc.)head poses. As a result, to assist the user in moving the reticle 1356,the wearable system may accelerate the movement (e.g., adjust an anglerelative to a reference head pose vector or adjust the position withinthe FOV) of the reticle 1356 in a direction corresponding to a directionto which the user's head is moving. By accelerating the movement of thereticle 1356, the wearable system advantageously reduces a degree towhich the user must bend, twist or crane his or her neck to align thereticle 1356 and target object, thereby reducing a likelihood of neckstrain or discomfort.

Returning to FIG. 13A, the reticle 1356 within FOV 1350 a is at a fixedlocation at the center 1360 of the FOV 1350 a. As the user tilts his orher head toward the ceiling or the floor, the wearable system may makereal-time determinations of the user's head pose. If the user's headpose begins to correspond to any of a range of undesired, uncomfortable,or straining head poses, the wearable system may begin to accelerate thereticle 1356.

For example, as illustrated in FIG. 13A, the user's real time head posecan be compared with one or more threshold head poses (e.g., max headpitch threshold 1394, min head pitch threshold 1396) to determinewhether the user's head pose corresponds to or falls within one or moreof a range of desired or undesired head poses. The wearable system 1320can determine the user's head pose by calculating one or more anglescorresponding to the degree at which the user is bending, turning,tilting, or rotating his or her neck. For example, the angulardifference between a vector corresponding to the user's current headpose and a reference head pose vector can be calculated. If the one ormore angles correspond to angles associated with undesired head poses,the wearable system may accelerate the movement of the reticle 1356, forinstance, by adding or removing an offset to an angle associated withthe reticle to adjust the reticle position within the FOV.

Examples of the addition of the offset are illustrated in FIG. 13A andFIG. 13B. As a first example, and continuing with the example where theuser's initial head pose corresponds to FOV 1350 a, the user must tilthis or her head down towards the ground to perceive FOV 1350 b. When theuser's head pose is at an angle such that the user perceives the FOV1350 b, the system can determine an angle 1386 between the user's newhead pose and a reference head pose (e.g., the user's initial head pose1348). As illustrated in FIG. 13A, the user's new head pose, whichcorresponds to the center 1360 of FOV 1350 b, is below (or does notsatisfy) a minimum head pitch threshold 1396. Based on the user's newhead pose, the system may determine that the user is looking down and/orthe user's head or neck is in, or is headed towards, an uncomfortable orotherwise undesirable pose. Accordingly, the system may accelerate thereticle 1356 in a downward direction such that the position is no longerin the default reticle position (e.g., the center) of the FOV. To makethis adjustment to the reticle position, the system may add an offset1364 to the angle 1386 at which the reticle is positioned relative tothe reference head pose. The offset (e.g., about 12 degrees) can beadded to the angle 1386 using various methods known in the art. Forexample, the offset 1364 can be added to or subtracted from the angle1386 to make the new angle more negative. Similarly, the offset canincrease or decrease the absolute value of the new angle.

As another example, and continuing with the assumption that user'sinitial head pose corresponds to FOV 1350 a, the user must tilt his orher head up towards the ceiling to perceive FOV 1350 c. When the user'shead pose is at an angle such that the user perceives the FOV 1350 c,the system can determine an angle 1390 between the user's new head poseand a reference head pose (e.g., the user's initial head pose 1348). Asillustrated in FIG. 13A, the user's new head pose, which corresponds tothe center 1360 of FOV 1350 c, is above (or satisfies) a maximum headpitch threshold 1394. Based on the user's new head pose, the system maydetermine that the user is looking up and/or the user's head or neck isin, or is headed towards, an uncomfortable or otherwise undesirablepose. Accordingly, the system may accelerate the reticle 1356 in anupward direction such that the position is no longer in the defaultreticle position relative to the FOV (e.g., the center of the FOV). Tomake this adjustment to the reticle position, the system may add anoffset 1368 to the angle 1390 at which the reticle is positionedrelative to the reference head pose. The offset (e.g., about 8 degrees)can be added to the angle 1390 using various methods known in the art.For example, the offset 1368 can be added to or subtracted from theangle 1390 to make the new angle more positive. Similarly, the offsetcan increase or decrease the absolute value of the new angle.

As another example, similar to FIG. 13A, FIG. 13B illustrates three FOVs1350 d, 1350 e, 1350 f which are schematically represented byrectangles, and which can correspond to a portion of the user's FOR 1310that is observed at a distinct period in time. In each of the threeillustrated scenes 1350 d, 1350 e, 1350 f, a user can perceive a reticle1356 via the display 1320. Initially, the reticle 1356 is at a fixedlocation at the center 1360 of the FOV 1350 d. As the user turns his orher head to the right, the wearable system 1320 may make real-timedeterminations of the user's head pose. As the user's head pose beginsto move toward any of a range of undesired, uncomfortable, or straininghead poses, the wearable system may begin to accelerate the reticle1356, thereby moving the reticle from the fixed location within the FOV.

As a non-limiting example, the user 1304 may desire to view or interactwith object 1384. For simplicity, the initial pose of the user's head1348 corresponds to the reference head pose, as described herein, andthe initial FOV perceived by the user is FOV 1350 d. As the user's headpose changes, the head pose can be compared with one or more thresholdshead poses (e.g., max head yaw threshold 1374, easing threshold 1372,min head yaw threshold, etc.) to determine whether the head pose passesa threshold and the reticle position should be accelerated.

Returning to the example, the user may begin to scan the FOR 1310 tosearch for object 1384. As the user turns his or her head to the right,the user's FOV correspondingly adjusts such that the user may eventuallyperceive FOV 1350 e. While perceiving FOV 1350 e, the user's head pose(which may correspond to the center of 1360 of the FOV 1350 e) satisfieseasing threshold 1372. Accordingly, the system may accelerate theposition of the reticle 1356 such that it is positioned slightly rightof the center 1360 of the FOV 1350 e. For example, the system may add anoffset 1376 (e.g., corresponding to an easing function) to the angle1392 corresponding to the reticle 1356 in FOV 1350 e.

The user may continue to scan the FOR 1310 to search for object 1384,for instance, by turning his or her head more to the right. At somelater point in time, the user may adjust his or her head pose such thatthe user perceives FOV 1350 f. As illustrated, the object 1384 isperceivable in FOV 1350 f, and it may be advantageous for the system toaccelerate the reticle, for example, toward the object 1384, to make iteasier for the user to select the object. Accordingly, system maydetermine that the user's head pose (which may correspond to the centerof 1360 of the FOV 1350 f) satisfies a maximum head yaw threshold 1374.As such, the system may accelerate the position of the reticle 1356 suchthat it is positioned even more right of the center of the FOV 1350 ethan the reticle positioned in FOV 1350 e. For example, the system mayadd a larger offset 1378 to the angle 1388 corresponding to the reticlein FOV 1350 f.

In some instances, in order to determine an amount of cursoracceleration in the horizontal direction, the system determines a poseof the user's head relative to the user's body, torso, shoulders, etc.For example, more neck strain may be more likely if a head pose vectoris more offset from a body pose vector. Accordingly, the system mayaccelerate a reticle at a relatively faster or slower rate depending onthe alignment of the user's head, neck, shoulders, body, etc.

In some cases, an additional inertial measurement unit (IMU) configuredto track a body, torso, or shoulder's pose of the user can be added andthe system can determine an angle or position of the user's headrelative to the user's body. The additional IMU can include an auxiliarydevice on the shoulders, chest or waist of the patient such as a pin, anecklace, a backpack, a belt pack, a totem, etc. In some embodiments,the system can include an external or user-facing sensor or camera thatcan, for instance, use computer vision processing to calculate a bodyvector.

Although the examples illustrated in FIGS. 13A and 13B correspond to auser tilting his or her head up to down and turning his or her head tothe right, this is not a limitation, and diagonal head movements (e.g.,combinations of vertical and horizontal movements) can be measured andthe reticle accelerated to a corner of the FOV if a diagonal threshold(or combination of vertical and horizontal thresholds) is passed.Further, the user can move his body, head, or eyes to perceivesubstantially any direction in space. In some embodiments, similar toadjusting the user's FOV based on a change in head pose, the system mayadjust a position or location of a reticle based on a change in eye pose(e.g., eye gaze). The eye pose can be determined by the inward facingimaging system 504 (shown in FIG. 4). The system can determine, usingone or more thresholds, whether the user's eyes may be strained and canalter a position of the reticle based on that determination.

Example Head Pose Angle Adjustments

FIG. 14 illustrates examples of adjusting position of a reticle based onthe user's head pose. The wearable system, such as wearable system 200of FIG. 2, can determine a user's head pose, for example, by determininga head pose vector corresponding to the user's head pose. Based on anangular difference between a head pose vector and a reference vector,the system can adjust a position of a reticle relative to a user's FOV.As described herein, the wearable system can include one or more headpose sensors such as, e.g., an IMU, which can be used to determine headpose, or an eye-tracking camera, which can be used to determine eyegaze. The wearable system can use data from such sensors to determinethe poses and angles described herein.

A head pose vector can provide an indication of the user's head pose.For example, a head pose vector can illustrate where the user is gazing,how the user's head is oriented, how the user's neck is bent, etc. Insome embodiments, the head pose vector can be defined as a vectorextending orthogonally from a coronal plane (e.g., frontal plane thatdivides the user's head into ventral and dorsal sections) of the user'shead. For example, the head pose plane may be a plane which is parallel(or substantially parallel) to the user's forehead. In some embodiments,the head pose plane can be parallel to the coronal plane, and cancomprise a line connecting the user's eyes or other facial features. Insome embodiments, the head pose vector can orthogonally extend from thecoronal plane (or a plane parallel to the coronal plane) from a centralpoint on the user's head such as a center of the user's forehead, thecenter of the user's eye, the user's nose, etc. In some embodiments, thehead pose vector can extend from any other point corresponding to theuser's head or neck. Accordingly, as the user's head pose changes, thehead pose vector also changes. As a few examples, FIG. 14 illustratesvarious example head pose vectors 1408, 1412, 1416, 1420, 1424, 1428.

A reference vector may be a subset of the potential head pose vectorsand may be used as a reference to determine an angle at which the user'shead is tilted or turned. The reference vector can be a vectorcorresponding to any head pose. For example, when considering thecoordinate values in the x-y-z coordinate system shown in FIG. 14, thereference vector may be equivalent to a head pose vector having avertical or y-component of zero (sometimes termed a level head posevector 1416). In some cases, the reference vector is determined byidentifying a vector in a horizontal plane that is perpendicular to aplane of the display. In some embodiments, the reference vector may bethe head pose vector when the user's head is in a natural resting state(for example, as a neutral head pitch vector 1420). In some embodiments,the level head pose vector corresponds to a position of the user's headwhen each of the coronal plane of the user's head (e.g., vertical planethat divides the body into belly and back sections), the sagittal planeof the user's head (e.g., an anatomical plane which divides the bodyinto right and left parts), and the axial plane of the user's head(e.g., a plane that divides the body into superior and inferior parts,roughly perpendicular to spine) of the user's head are orthogonal to oneanother. As illustrated in FIG. 14, in some instance the angulardifference between the neutral head pitch vector 1420 and the level headpose vector 1416 is approximately −20 degrees (e.g., the neutral headpitch vector 1420 is approximately 20 degrees below the level head posevector 1416). In some applications, a user may look to a particulardirection more frequently, and the system may adjust the reference headpose to reduce or ameliorate the neck strain. For example, a gamerelating to shooting flying birds and a star gazing application thatteaches users about constellations may both involve a user periodicallyor constantly looking up. Similarly, a game of chess may involve a userperiodically or constantly looking down. Other games or applications mayinvolve a user periodically or constantly looking left, right,diagonally, or the like. Accordingly, in some cases, the reference headpose can be updated to hedge toward a head pose associated with thismore frequent direction. For example, the reference head pose can be anaverage or most-common head pose of a user over a particular period oftime or an average or most-common head pose of a user while the useruses the particular application or plays the particular game. Byconfiguring the reference head pose to hedge toward or match a head poseassociated with a common head pose of the user, the system can reduce orameliorate the neck strain.

As described herein, the wearable system can identify a range of headpositions or a range of head orientations, which together serve tospecify one or more ranges of head poses that may correspond touncomfortable or straining head poses. The bounds of such ranges may beseen as corresponding to thresholds. For example, if a user tilts orturns his or head too far in one direction, the user's head pose mayfail to satisfy (e.g., be less than) a minimum head pitch, head yaw, orhead roll threshold. Similarly, if the user tilts or turns his or herhead too far in another direction, the user's head pose may satisfy(e.g., be greater than) a maximum head pitch, had yaw, or head rollthreshold. In instances such as these, head poses which are less than(or fail to satisfy) a minimum head pose threshold and head poses whichare greater than (or satisfy) a maximum head pose threshold maycorrespond to one or more ranges of head tilting or turning which maycause neck strain or discomfort.

FIG. 14 illustrates three examples of an adjustment to a position, or anadjustment to an angle relative to a reference head pose, of thereticle. In a first example, the user's head pose corresponds to headpitch vector 1420 that, in this example, corresponds to a neutral headpose threshold. The neutral head pose threshold, as described herein,may correspond to a natural resting state of the head and may representa more comfortable head pose of the user than, say, a level head pose1416. Accordingly, in some instances, a neutral head pose vector may beused as a reference head pose vector instead of, or in addition to, alevel head pose vector.

Returning to the first example, a −20 degree angular difference existsbetween head pitch vector 1420 and the reference head pose vector 1416which, in this case, is also a level head pose vector 1416. Based on thedetermined angular difference of −20 degrees, the system determines thatthe user's head pose is above (e.g., satisfies) the minimum pitchthreshold of −45 degrees. In addition, the system determines that theangular difference of −20 degrees is below (e.g., does not satisfy) themaximum pitch threshold of 5 degrees. Based on these determinations, thesystem may determine not to adjust the position of the reticle.Accordingly, as seen in scene 1436, the reticle remains at a defaultlocation (e.g., the center) within the scene 1436.

In a second example, the user's head pose corresponds to head posevector 1412. As illustrated, a +5 degree angular difference existsbetween head pose vector 1412 and the reference head pose vector 1416.Based on the determined angular difference of +5 degrees, the systemdetermines that the user's pose is equal to (e.g., satisfies) a maximumpitch threshold of +5 degrees. Accordingly, the system adjusts an angleof the reticle by an offset amount, in this case 8 degrees. Thisadjustment can be seen in scene 1432, where the position of the reticleappears towards the top of scene 1432, rather than at the defaultposition (e.g., the center) of scene 1432.

In a third example, the user's head pose corresponds to head pose vector1424. As illustrated, a −45 degree angular difference exists betweenhead pose vector 1424 and the reference head pose vector 1416.Accordingly, based on the determined angular difference of −45 degrees,the system determines that the user's pose is equal to (e.g., does notsatisfy) a minimum head pitch threshold of −45 degrees. Accordingly, thesystem adjusts an angle of the reticle by an offset amount, in this case−12 degrees. This adjustment can be seen in scene 1440, where theposition of the reticle appears towards the bottom of scene 1440, ratherthan at the default position (e.g., the center) of scene 1440.

It should be noted that the examples shown in FIG. 14 are merelyillustrative and should not be construed as limiting. Accordingly, insome embodiments a reference vector other than the level head posevector 1416 is used to determine angular difference. Fewer, more, ordifferent thresholds can be used by the system to determine whether ahead position passes a threshold and to adjust a position of thereticle. Further, the values corresponding to the maximum head pitchthreshold 1412, the minimum head pitch threshold 1424, the neutral headpitch vector 1420, the offset amount corresponding to the maximum headpitch threshold 1412 and/or the offset amount corresponding to theminimum head pitch threshold 1424 are for purposes of illustration. Inother implementations, the maximum head pitch threshold 1412 can be in arange from 10 degrees below the level head pose vector 1416 to 25degrees above the level head pose vector 1416. The minimum head pitchthreshold 1424 can be in a range from 60 degrees below the level headpose vector 1416 to 30 degrees below the level head pose vector 1416.The neutral head pitch vector 1420 can be in a range from 30 degreesbelow the level head pose vector 1416 to level with the level head posevector 1416. The offset amount corresponding to the maximum pitchthreshold 1412 can be in a range of +1 degree to +25 degrees. The offsetamount corresponding to the minimum pitch threshold 1424 can be in arange of −1 degree to −25 degrees. In addition, the one or morethresholds may be satisfied in various ways. For instance, in somecases, a value equivalent to the threshold value will satisfy thethreshold, while, in other cases, a value equivalent to the thresholdvalue will not satisfy the threshold.

Example Head Pose Angle Adjustments to the Position of a Reticle

FIG. 15 demonstrates example relationships between a user's head poseand an adjustment to position of a reticle. The graph 1500 illustratesexample reticle angle adjustments versus an angular difference. Asdescribed herein, angular difference (e.g., the x-axis) can be definedas the difference in angle between a head pose vector and a referencevector (e.g., a level head pose vector, a neutral head pose vector,etc.) In addition, an adjustment to the reticle angle (e.g., the y-axis)can correspond to how the system changes the position of the reticlebased on determined head pose.

Graph 1500 illustrates several different possible relationships (e.g.,lines 1560, 1564, 1568) between the adjustment to the reticle angle andthe angular difference. For example, lines 1560, 1568 illustrate agradual adjustment to the reticle angle as the angular differencechanges, while line 1564 illustrates no adjustment to the reticle anglewithin a specified range (e.g., −30 degrees to −10 degrees) and then agradual adjustment to the reticle angle as the angular differenceextends outside of the specified range (e.g., below a minimum head posethreshold, above a maximum head pose threshold).

As illustrated by each of the relationships 1564, 1560, and 1568, if theangular difference is 5 degrees or higher (e.g., the user is looking upat about 5 degrees or more above a level head pose vector), then thesystem determines that the user's head pose is greater than or equal to(e.g., satisfies) a maximum head pitch threshold (e.g., +5 degrees).Accordingly, the system adjusts an angle of the reticle by a maximumoffset amount of 8 degrees. In addition, if the angular difference isless than or equal to −45 degrees (e.g., the user is looking down about45 degrees or more below a level head pose vector), then the systemdetermines that the user's head pose is less than or equal to (e.g.,does not satisfy) a minimum head pitch threshold (e.g., −45 degrees).Accordingly, the system adjusts an angle of the reticle by a minimumoffset amount of −12 degrees.

Furthermore, if the angular difference is between about −45-degrees andabout 5 degrees (e.g., between the minimum head pose threshold and themaximum head pose threshold), then the system determines that the user'shead pose is greater than (e.g., satisfies) a minimum head pitchthreshold and is less than (e.g., does not satisfy) a maximum head pitchthreshold. Accordingly, the system adjusts an angle of the reticle by anoffset amount, which can be determined, for example, by various linear,exponential, piecewise, or easing functions. For example, an easingfunction can provide progressively increasing or decreasing reticleangle adjustments.

Example Processes of Reticle Positioning Reticle Positioning orAdjustment Based on Head Pose

FIG. 16 illustrates a flowchart for an example reticle positionadjustment process 1600. The example process 1600 may be performed byone or more components of the wearable system 200 such as, e.g., theremote processing module 270 or the local processing and data module260, alone or in combination. The display 220 of the wearable system 200can present reticle(s) to the user, the inward-facing imaging system 462can obtain the eye images for eye gaze determination, and IMUs,accelerometers, or gyroscopes can determine head pose.

At block 1610, the wearable system can receive data indicating theuser's current head pose. A head pose can describe a position and anorientation of the user's head. The data can include the currentposition and orientation of the user's head or the movements of theuser's head in the 3D space. The position may be represented bytranslational coordinate values (such as, e.g., coordinate values in anx-y-z coordinate system shown in FIG. 6). For example, as the user'shead tilts or turns, the wearable system can track and record the user'shead movements. The orientation may be represented by vectors or angularvalues relative to a natural resting state of the head. For example, thevectors or angular values can represent the head tilting forward andbackward (e.g., pitching), turning left and right (e.g., yawing), andtilting side to side (e.g., rolling).

The wearable system can identify a range of head positions or a range ofhead orientations, which together serve to specify one or more ranges ofhead poses which may correspond to uncomfortable or straining headposes. The bounds of such ranges may be seen as corresponding tothresholds. For example, a maximum or minimum head pitch, head yaw, orhead roll threshold may correspond to one or more ranges of head tiltingor turning left which may cause neck strain or discomfort. Similarly, aneutral or reference head pose threshold may correspond to a naturalresting state of the head.

The head poses that fall within these ranges can correspond to headposes in which an adjustment to a position of reticle may be desired.For example, the wearable system can render a reticle in 3D space for auser. The reticle may be rendered in a rig space (which may berepresented by a coordinate system with respect to an HMD. The reticlemay be represented in a variety of graphical forms, which may include1D, 2D, and 3D images.). The reticle may correspond to the user'scurrent position with respect to the user's field of view, and mayrepresent, for example, the user's direction of gaze. When the usermoves around, the reticle may also move with the user. As the user'shead pose changes, for example such that it falls within one of theaforementioned ranges, the position of the reticle may be adjusted toprevent, ease or lessen neck strain or discomfort.

The wearable system can track the head poses using one or more sensorsinternal to an HMD such as, e.g., an IMU or an outward-facing imagingsystem (e.g., to track a reflected image of the user's head) or externalto the HMD (such as, e.g., a camera mounted to a wall in the user'sroom).

At block 1620, the wearable system can identify or determine, based atleast in part on the head pose data acquired from block 1610, areal-time head pose of the user. For example, the wearable system canidentify or determine a head pose vector, as described herein, whichcorresponds to the user's head pose. In some cases, the head pose vectoris determined using an AR software development kit (SDK).

At block 1630, the wearable system can identify, access, or determine areference head pose of the user. For example, the reference head posecan correspond to a neutral head pose and may be based at least in parton the head pose data acquired from block 1610. The wearable system candetermine a reference head pose vector, as described herein, which cancorrespond to a neutral head pose, such as the head pose relative to anatural resting state of the head. In some cases, the reference headpose vector is determined using an AR SDK. In some implementations, thereference or neutral head pose of the user is set at a default value(e.g., at an angle −20 degrees), and the wearable system determines thereference head pose by accessing the default value (e.g., by queryingthe AR SDK).

At block 1640, the wearable system can determine, based at least in parton the a comparison between the current head pose and the reference headpose, an adjustment for a position of a virtual reticle projected on ahead mounted display. For example, the system can determine a valueindicative of a difference between the head pose determined from block1620 and the reference head pose determined from block 1630. Forexample, an angular difference between the head pose and the referencehead pose can be calculated with respect to coordinate values (such as,e.g., coordinate values in an x-y-z coordinate system shown in FIG. 6).In some embodiments, the wearable system can determine the angulardifference between the head pose vector and the reference head posevector. For example, as shown in FIG. 14, the angular difference can beused to determine an angle that corresponds to a degree of tilting orturning of the user's head with respect to a reference head pose. Basedat least in part on the angular difference, the wearable system candetermine whether the user is looking up or down, or otherwise tiltingor turning his head. In some cases, the angular difference is determinedusing an AR SDK.

The wearable system can determine an adjusted reticle position based atleast in part on the pose of the user's head. For example, the wearablesystem can determine whether the user has assumed a head pose that fallswithin one or more of the identified ranges of head poses. The wearablesystem can determine whether the user's head pose is at a position ororientation that can result in an adjustment of the position of thereticle. As an example, the wearable system can determine whether theuser's head pose falls within the identified range of head positions andthe identified range of head orientations. The wearable system may makesuch a determination by comparing the user's head pose with thresholdvalues that define the bounds of the identified range of head positions(e.g., translational coordinate values), or by comparing the headorientation (e.g., angular difference) associated with the user's headpose with threshold values that define the bounds of the identifiedrange of head orientations (e.g., angular values).

With reference to FIG. 14, based on a determination that the user hasassumed a head pose corresponding to head pose vector 1412, the wearablesystem can determine that the angular difference (e.g., differencebetween reference head pose vector) is 5 degrees. Accordingly, bycomparing the angular difference with threshold values (e.g., a neutralhead pose threshold corresponding to vector 1416), the wearable systemcan determine if an adjustment of the position of the reticle isdesired. For example, returning to the example of FIG. 14, the wearablesystem may determine that the angular difference of 5 degreescorresponding to head pose vector 1412 satisfies a maximum head posethreshold. The wearable system may then adjust the position of thereticle such that a head pose vector 1408 pointing to the newly adjustedreticle has an angular difference of 8 degrees. Accordingly, thewearable system can accelerate or decelerate reticle movement based atleast in part on head pose or changes in head pose.

The various blocks described herein can be implemented in a variety oforders, and that the wearable system can implement one or more of theblocks concurrently and/or change the order, as desired. Fewer, more, ordifferent blocks can be used as part of the process 1600. For example,the process 1600 can include blocks for determining a position of areticle, providing an indication that the position of the reticle wasupdated, etc.

Furthermore, although process 1600 has been logically associated withpreventing or reducing a likelihood of neck strain, similar techniquescan be utilized to prevent or reduce a likelihood of eyestrain. Forexample, the system can obtain eye gaze data and from the eye gaze datacan determine an eye gaze vector and a neutral eye gaze vector. Thesystem can further determine an angular difference between the eye gazevector and the neutral eye gaze vector and, based on the angulardifference and one or more thresholds, can determine an adjustment forat least one of a position of the virtual reticle or a 3D view of thedisplay.

Reticle Positioning or Adjustment Based on Angular Difference

FIG. 17 illustrates a flowchart for an example reticle adjustmentprocess 1700. The example process 1700 may be performed by one or morecomponents of the wearable system 200 such as, e.g., the remoteprocessing module 270 or the local processing and data module 260, aloneor in combination. The display 220 of the wearable system 200 canpresent reticle(s) to the user, the inward-facing imaging system 462 canobtain the eye images for eye gaze determination, and IMUs,accelerometers, or gyroscopes can determine head pose.

The wearable system can render a reticle in 3D space for a user. Thereticle may be rendered in a rig space (which may be represented by acoordinate system with respect to an HMD. The reticle may be representedin a variety of graphical forms, which may include 1D, 2D, and 3Dimages.). The reticle may correspond to the user's current position withrespect to the user's field of view, and may represent, for example, theuser's direction of gaze. When the user moves around, the reticle mayalso move with the user. In addition, the reticle may point at one ormore objects, and the user may select a target object to which thereticle is pointing. In some instances, the position of the reticlewithin the user's FOV may remain constant as the user's head moves. Forexample, the reticle may be positioned in the center of the user's fieldof view and will stay at the center, even as the user's FOV changes.Accordingly, in some cases, to select a target object with the reticle,the user must move his or her head to adjust the FOV such that thecenter of the FOV (e.g., the location of the reticle) is on the targetobject.

However, it may be advantageous for the wearable system to adjust aposition of the reticle within the user's FOV. For example, in instanceswhere the user desires to select a target object that is high in the air(e.g., the ceiling), low to the ground (e.g., the floor), far to theright, or far to the left, the wearable system can adjust the positionof the reticle within the user's FOV to help the user position thereticle at the location of the target object, but without requiring theuser to tilt or turn his or her head such that the center of the user'sFOV is on the target object. Accordingly, the wearable system mayimplement process 1600 to adjust a position of a reticle.

At block 1710, the wearable system can identify or determine one or moreranges of head positions or ranges of head orientations, which togetherserve to specify one or more ranges of head poses which may correspondto uncomfortable or straining head poses. The bounds of such ranges maybe seen as corresponding to thresholds. For example, the wearable systemcan identify or determine a maximum head pitch threshold and/or aminimum head pitch threshold that may correspond to a degree or angle offorward or backward head tilting that will results in maximum or minimumreticle angle adjustment. In some embodiments, the wearable system canidentify or determine a maximum head yaw threshold or a minimum head yawthreshold which may correspond to a degree or angle of left or righthead turning before a user's neck is in an uncomfortable or strainingposition. Similarly, the wearable system can identify or determine amaximum head roll threshold or a minimum head roll threshold which maycorrespond to a degree or angle of side to side head tilting before auser's neck is in an uncomfortable or straining position. Furthermore,the wearable system can identify or determine a neutral head pitchthreshold, a neutral head yaw threshold, or a neutral head rollthreshold which may correspond to a natural resting state of the head.In some cases, these thresholds may be user inputs, may be determinedduring a calibration stage, or may be constants within the system.

At blocks 1720 and 1730, the wearable system can determine, based atleast in part on received head pose data, a head pose vector andreference (or neutral) head pose vector, as described herein.

At block 1740, the wearable system can determine an angular differencebetween the head pose vector and the neutral head pose vector. Forexample, as shown in FIG. 14, the angular difference can be used todetermine an angle that corresponds to a degree of tilting or turning ofthe user's head with respect to a level head pose vector. Based at leastin part on the angular difference, the wearable system can determinewhether the user is looking up or down, or otherwise tilting or turninghis head.

At block 1750, the wearable system can calculate a reticle positionadjustment. Reticle position adjustments (e.g., angle adjustments oroffsets) can be associated with each of the thresholds to help, forinstance, accelerate cursor movement. For example, if the head posesatisfies a maximum head pitch threshold, maximum head yaw threshold, ora maximum head roll threshold, the wearable system may adjust an angleor position of the reticle by a first predetermined amount (e.g., +8degrees). Similarly, if the head pose does not satisfy a minimum headpitch threshold, minimum head yaw threshold, or a minimum head rollthreshold, the wearable system may adjust an angle or position of thereticle by a second predetermined amount (e.g., −12 degrees). In someembodiments, if the head pose satisfies a minimum threshold (e.g.,pitch, yaw, or roll) and does not satisfy a maximum threshold (e.g.,pitch, yaw, or roll), the wearable system may adjust an angle orposition of the reticle using an easing function. For example, if thehead pose satisfies a minimum head pitch threshold and does not satisfya neutral head pitch threshold, the wearable system may adjust an angleor position of the reticle using the following equation 1:

$\begin{matrix}{{VPA} = {\max{HPA}*{{ease}\left( \frac{P - {neutralHP}}{{\max{HPA}} - {neutralHP}} \right)}}} & \left( {{Equation}1} \right)\end{matrix}$

where VPA is a vertical angular pitch adjustment, maxHPA is a maximumhead pitch adjustment, P is the angle between a head pose vector an alevel head pose vector, neutralHP is an angle corresponding to theneutral head pose vector, and ease( ) is an easing function such aseaseOutSine.

Similarly, if the head pose satisfies a neutral head pitch threshold anddoes not satisfy a maximum head pitch threshold, the wearable system mayadjust an angle or position of the reticle using the following equation2:

$\begin{matrix}{{VPA} = {\min{HPA}*{{ease}\left( \frac{P - {\min{HPA}}}{{neutralHP} - {\min HP}} \right)}}} & \left( {{Equation}2} \right)\end{matrix}$

where VPA is a vertical angular pitch adjustment, minHPA is a minimumhead pitch adjustment, P is the angle between a head pose vector an alevel head pose vector, neutralHP is an angle corresponding to theneutral head pose vector, and ease( )is an easing function such aseaseOutSine.

The various blocks described herein can be implemented in a variety oforders, and that the wearable system can implement one or more of theblocks concurrently and/or change the order, as desired. Fewer, more, ordifferent blocks can be used as part of the process 1700. For example,the process 1700 can include blocks for determining a position of areticle, providing an indication that the position of the reticle wasupdated, etc.

Furthermore, although process 1700 has been logically associated withpreventing or reducing a likelihood of neck strain, similar techniquescan be utilized to prevent or reduce a likelihood of eye strain. Forexample, the system can obtain eye gaze data and from the eye gaze datacan determine an eye gaze vector and a neutral eye gaze vector. Thesystem can further determine an angular difference between the eye gazevector and the neutral eye gaze vector and, based on the angulardifference and one or more thresholds, can determine an adjustment forat least one of a position of the virtual reticle or a 3D view of thedisplay.

Reticle Positioning or Adjustment Based on Head Pitch Thresholds

FIG. 18 illustrates a flowchart for an example reticle adjustmentprocess. The example process 1800 may be performed by one or morecomponents of the wearable system 200 such as, e.g., the remoteprocessing module 270 or the local processing and data module 260, aloneor in combination. The display 220 of the wearable system 200 canpresent reticle(s) to the user, the inward-facing imaging system 462 canobtain the eye images for eye gaze determination, and IMUs,accelerometers, or gyroscopes can determine head pose.

At block 1810, similar to block 1740 of FIG. 17, the wearable systemcalculates an angular difference between a head pose vector and aneutral head pose vector.

At block 1820, the wearable system determines whether the angulardifference satisfies or fails to satisfy a minimum head pitch threshold.As described herein, head pitch may correspond to tilting the headforward or backward. Accordingly, the angular difference may fail tosatisfy a minimum head pitch threshold when the user is straining orbending his or her neck forward (e.g., to look at the ground). Forexample, the minimum head pitch threshold may correspond to an angulardifference of about −30, −35, −40, −45, −50, 55, or −60 degrees (+/− afew degrees). Thus, in some cases, if the angular difference is at orbelow the minimum head pitch threshold, the angular difference fails tosatisfy the minimum head pitch threshold. However, in some cases, theangular difference satisfies the minimum head pitch threshold if it isat or below the minimum head pitch threshold.

It should be noted that if the angle between the head pose vector andthe neutral head pose vector (e.g., head pose vector minus neutral headpose vector) is positive, then the user is looking up relative to aneutral position. Likewise, if the angle is negative, the user islooking down. Accordingly, in some cases, the system can determine ifthe user is looking up or down, and can take an absolute value of theangular difference to ensure a positive angle. In some embodiments, thesystem can determine an angular difference (e.g., head pose vector minusneutral head pose vector) and will understand that a negative valueindicates that the user is looking down.

At block 1830, the wearable system determined that the angulardifference does not satisfy the minimum head pitch threshold. As such,the wearable system determines an adjusted reticle position using anangle adjustment associated with the minimum head pitch threshold. Insome cases, the angle adjustment can be about −30, −25, −20, −15, −12,−10, −5, or −2 degrees.

At block 1840, the wearable system determines whether the angulardifference satisfies or fails to satisfy a maximum head pitch threshold.As described herein, head pitch may correspond to tilting the headforward or backward. Accordingly, the angular difference may satisfy amaximum head pitch threshold when the user is straining or bending hisor her neck backward (e.g., to look at the sky). For example, themaximum head pitch threshold may correspond to an angular difference ofabout 2, 5, 8, 10, 12, 15, 20, 25, or 30 degrees (+/− a few degrees).Thus, in some cases, if the angular difference is at or above themaximum head pitch threshold, the angular difference satisfies themaximum head pitch threshold. However, in some cases, the angulardifference fails to satisfy the maximum head pitch threshold if it is ator above the maximum head pitch threshold.

At block 1850, the wearable system determined that the angulardifference satisfies the maximum head pitch threshold. As such, thewearable system determines an adjusted reticle position using an angleadjustment associated with the maximum head pitch threshold. In somecases, the angle adjustment can be about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,12, 15, or 20 degrees.

At block 1860, the wearable system determines whether the angulardifference satisfies or fails to satisfy a neutral head pitch threshold.As described herein, head pitch may correspond to tilting the headforward or backward. The neutral head pitch threshold may correspond toa natural resting state of the head. For example, the neutral head pitchthreshold may correspond to an angular difference of about −10, −15,−20, −25, or −30 degrees (+/− a few degrees). Thus, in some cases, ifthe angular difference is at or above the neutral head pitch threshold,the angular difference satisfies the neutral head pitch threshold.Similarly, if the angular difference is below the neutral head pitchthreshold, the angular difference may not satisfy the neutral head pitchthreshold. However, in some cases, the angular difference fails tosatisfy the neutral head pitch threshold if it is at or above theneutral head pitch threshold and may satisfy the neutral head pitchthreshold it is below the neutral head pitch threshold.

At block 1870, the wearable system determined that the angulardifference satisfies the neutral head pitch threshold. As such, thewearable system may adjust an angle or position of the reticle usingequation 2 above.

At block 1880, the wearable system determined that the angulardifference does not satisfy the neutral head pitch threshold. As such,the wearable system may adjust an angle or position of the reticle usingequation 1 above.

The various blocks described herein can be implemented in a variety oforders, and that the wearable system can implement one or more of theblocks concurrently and/or change the order, as desired. Fewer, more, ordifferent blocks can be used as part of the process 1800. For example,the process 1800 can include blocks for determining additionalthresholds, providing an indication that the position of the reticle wasupdated, etc.

Furthermore, although process 1800 has been logically associated withpreventing or reducing a likelihood of neck strain, similar techniquescan be utilized to prevent or reduce a likelihood of eye strain. Forexample, the system can obtain eye gaze data and from the eye gaze datacan determine an eye gaze vector and a neutral eye gaze vector. Thesystem can further determine an angular difference between the eye gazevector and the neutral eye gaze vector and, based on the angulardifference and one or more thresholds, can determine an adjustment forat least one of a position of the virtual reticle or a 3D view of thedisplay.

Example Software Code

Appendix A includes an example of code in the C# programming languagethat calculates head pose and vertical pitch adjustment. The codeimplements an example of the reticle adjustment process 1700 describedwith reference to FIG. 17. Appendix A is hereby incorporated byreference herein in its entirety so as to form a part of thisspecification.

Additional Aspects

In a first aspect, a system comprising: a head pose sensor configured toobtain head pose data of a user of the system; non-transitory memoryconfigured to store the head pose data; a display configured to bepositioned in front of an eye of a user, the display configured toproject a virtual reticle toward the eye of the user; a hardwareprocessor in communication with the head pose sensor, the display, andthe non-transitory memory, the hardware processor programmed to: obtainthe head pose data of the user; identify a head pose of the user basedat least in part on the head pose data; determine an adjustment for aposition of the virtual reticle based at least in part on a comparisonbetween the head pose of the user and a reference head pose, and causethe virtual reticle to change in position based at least in part on thedetermined adjustment.

In a second aspect, the system of aspect 1, wherein the virtual reticlecomprises a movable indicator identifying a position of the user withinin a field of view of the user.

In a third aspect, the system of aspect 1 or aspect 2, wherein head posedata corresponds to at least one of an indication of a yaw, a pitch, ora roll of a head of the user.

In a fourth aspect, the system of aspect 3, wherein the indication of ayaw, pitch, or roll is with respect to the reference head pose.

In a fifth aspect, the system of aspect 4, wherein the reference headpose corresponds to a level head pose of the head of the user.

In a sixth aspect, the system of aspect 5, wherein the level head posecomprises a head pose in which a coronal plane of the head of the user,a sagittal plane of the head of the user, and an axial plane of the headof the user are each orthogonal to one another.

In a seventh aspect, the system of any one of aspects 1 to 6, whereinthe reference head pose comprises a head pose corresponding to a naturalresting state of the head of the user.

In an eighth aspect, the system of aspect 7, wherein the natural restingstate of the head of the user corresponds to between −5 to 5 degrees ofyaw, between −5 to 5 degrees of roll, and between −15 to −25 degrees ofpitch, relative to the level head pose.

In a ninth aspect, the system of aspect 8, wherein the reference headpose corresponds to at least one of 0 degrees of yaw, 0 degrees of roll,or −20 degrees of pitch, relative to the level head pose.

In a tenth aspect, the system of any one of aspects 1 to 9, wherein thehardware processor is further programmed to identify a head pose vectorcorresponding to the head pose of the user and identify a reference headpose vector corresponding to the reference head pose.

In an 11th aspect, the system of aspect 10, wherein the hardwareprocessor is further programmed to determine an angular differencebetween the head pose vector and the reference head pose vector based atleast in part on the comparison between the head pose of the user andthe reference head pose, wherein the angular difference corresponds toat least one of a difference in yaw, pitch, or roll of the head pose ofthe user with respect to the reference head pose.

In a 12th aspect, the system of aspect 11, wherein to determine theadjustment for the position of the virtual reticle, the hardwareprocessor is programmed to compare the determined angular difference toone or more head pose thresholds.

In a 13th aspect, the system of aspect 12, wherein the one or more headpose thresholds comprises at least one of a maximum head pose thresholdor a minimum head pose threshold.

In a 14th aspect, the system of aspect 13, wherein the maximum head posethreshold corresponds to at least one of a maximum head yaw threshold, amaximum head roll threshold, or a maximum head pitch threshold.

In a 15th aspect, the system of aspect 14, wherein the maximum head yawthreshold is 50 degrees, the maximum head roll threshold is 20 degrees,or the maximum head pitch threshold is 5 degrees, relative to thereference head pose.

In a 16th aspect, the system of any one of aspects 13 to 15, wherein theminimum head pose threshold corresponds to at least one of a minimumhead yaw threshold, a minimum head roll threshold, or a minimum headpitch threshold.

In a 17th aspect, the system of aspect 16, wherein the minimum head yawthreshold is −50 degrees, the minimum head roll threshold is −20degrees, or the minimum head pitch threshold is −45 degrees, relative tothe reference head pose.

In an 18th aspect, the system of any one of aspects 13 to 17, whereinthe hardware processor is further programmed to: responsive to adetermination that the angular difference fails to satisfy the minimumhead pose threshold, determine the adjustment for the position of thevirtual reticle based at least in part on a first adjustment value.

In a 19th aspect, the system of aspect 18, wherein the first adjustmentvalue is about −12 degrees.

In a 20th aspect, the system of any one of aspects 13 to 19, wherein thehardware processor is further programmed to, responsive to adetermination that the angular difference satisfies the maximum headpose threshold, determine the adjustment for the position of the virtualreticle based at least in part on a second adjustment value.

In a 21st aspect, the system of aspect 20, wherein the second adjustmentvalue is about +5 degrees.

In a 22nd aspect, the system of any one of aspects 13 to 21, wherein thehardware processor is further programmed to: responsive to adetermination that the angular difference satisfies the minimum headpose threshold and fails to satisfy the maximum head pose threshold,determine the adjustment for the position of the virtual reticle basedat least in part on a third adjustment value.

In a 23rd aspect, the system of aspect 22, wherein the third adjustmentvalue corresponds to an easing function.

In a 24th aspect, the system of aspect 22 or aspect 23, wherein thethird adjustment value is about 0 degrees.

In a 25th aspect, the system of any one of aspects 1 to 24, wherein tocause the virtual reticle to change in position comprises causing thevirtual reticle to change position from a default reticle position of afield of view of the user.

In a 26th aspect, the system of aspect 25, wherein the default reticleposition comprises a center of a field of view of the user.

In a 27th aspect, the system of any one of aspects 1 to 26, wherein thehead pose sensor comprises an inertial measurement unit (IMU), anaccelerometer, a gyroscope, or a magnetometer.

In a 28th aspect, the system of any one of aspects 1 to 27, wherein thewearable system comprises a head mounted wearable system.

In a 29th aspect, a method of adjusting a position of a virtual reticleidentifying a position of a user within a field of view corresponding toa display of a display system, the method comprising: obtaining headpose data of a user of a display system from a head pose sensorconfigured to track a head pose of the user; identifying a head posevector corresponding to the head pose of the user based at least in parton the head pose data; identifying a reference head pose vectorcorresponding to a reference head pose; determining an angulardifference between the head pose vector and the reference head posevector based at least in part on a comparison between the head pose ofthe user and the reference head pose, wherein the angular differencecorresponds to at least one of a difference in yaw, pitch, or roll ofthe head pose of the user with respect to the reference head pose;comparing the determined angular difference to one or more head posethresholds, wherein the one or more head pose thresholds comprises atleast one of a maximum head pose threshold or a minimum head posethreshold; responsive to a determination that the angular differencefails to satisfy the minimum head pose threshold, determining anadjustment for a position of a virtual reticle based at least in part ona first adjustment value, wherein the position of the virtual reticlecorresponds to a position of the movable indicator identifying theposition of the user within the field of view of the user; responsive toa determination that the angular difference satisfies the maximum headpose threshold, determining the adjustment for the position of thevirtual reticle based at least in part on a second adjustment value;responsive to a determination that the angular difference satisfies theminimum head pose threshold and fails to satisfy the maximum head posethreshold, determining the adjustment for the position of the virtualreticle based at least in part on a third adjustment value; and causingthe position of the virtual reticle to be adjusted from a defaultreticle position of the field of view of the user based on thedetermined adjustment. The method can be performed under control of thedisplay system, for example, by a hardware processor programmed toperform the operations of the method.

In a 30th aspect, the method of aspect 29, wherein the virtual reticlecomprises a movable indicator identifying the position of the userwithin the field of view of the user.

In a 31st aspect, the method of aspect 29 or aspect 30, wherein headpose data corresponds to at least one of an indication of yaw, pitch, orroll of the head of the user.

In a 32nd aspect, the method of aspect 31, wherein the indication ofyaw, pitch, or roll of the head of the user is with respect to thereference head pose.

In a 33rd aspect, the method of any one of aspects 29 to 32, wherein thereference head pose of the user corresponds to a level head pose of thehead of the user.

In a 34th aspect, the method of aspect 33, wherein the level head posecomprises a head pose in which a coronal plane of the head of the user,a sagittal plane of the head of the user, and an axial plane of the headof the user are each orthogonal to one another.

In a 35th aspect, the method of any one of aspects 29 to 34, wherein thereference head pose comprises a head pose corresponding to the head ofthe user in a natural resting state.

In a 36th aspect, the method of aspect 35, wherein the natural restingstate of the user's head corresponds to about −5 to 5 degrees of yaw,about −5 to 5 degrees of roll, or about −15 to -25 degrees of pitch,relative to a level head of the user.

In a 37th aspect, the method of aspect 35 or aspect 36, wherein thereference head pose corresponds to at least one of 0 degrees of yaw, 0degree of roll, or −20 degrees of pitch, relative to the level headpose.

In a 38th aspect, the method of any one of aspects 29 to 37, wherein themaximum head pose threshold corresponds to at least one of a maximumhead yaw threshold, a maximum head roll threshold, or a maximum headpitch threshold.

In a 39th aspect, the method of any one of aspects 29 to 38, wherein themaximum head yaw threshold is about 50 degrees, the maximum head rollthreshold is about 20 degrees, and the maximum head pitch threshold isabout 5 degrees, relative to the reference head pose.

In a 40th aspect, the method of any one of aspects 29 to 39, wherein theminimum head pose threshold corresponds to at least one of a minimumhead yaw threshold, a minimum head roll threshold, or a minimum headpitch threshold.

In a 41st aspect, the method of any one of aspects 29 to 40, wherein theminimum head yaw threshold is about −50 degrees, the minimum head rollthreshold is about −20 degrees, and the minimum head pitch threshold isabout −45 degrees, relative to the reference head pose.

In a 42nd aspect, the method of any one of aspects 29 to 41, wherein thefirst adjustment value is about −12 degrees.

In a 43rd aspect, the method of any one of aspects 29 to 42, wherein thesecond adjustment value is about +5 degrees.

In a 44th aspect, the method of any one of aspects 29 to 43, wherein thethird adjustment value corresponds to an easing function.

In a 45th aspect, the method of any one of aspects 29 to 44, wherein thethird adjustment value is about 0 degrees.

In a 46th aspect, the method of any one of aspects 29 to 45, wherein thedefault reticle position comprises the center of the user's field ofview.

In a 47th aspect, the method of any one of aspects 29 to 46, wherein thehead pose sensor comprises an inertial measurement unit (IMU), anaccelerometer, a gyroscope, or a magnetometer.

In a 48th aspect, a method of adjusting a position of a virtual reticleidentifying a position of the user within a field of view correspondingto a display of a display system, the method comprising: obtaining headpose data of the user of the display system; identifying a head posebased at least in part on the head pose data; identifying a referencehead pose; determining, based at least in part on a comparison betweenthe head pose and the reference head pose, an adjustment for a positionof a virtual reticle projected on a head mounted display.

In a 49th aspect, the method of aspect 48, further comprising obtainingthe head pose data of the user of the display system from a head posesensor configured to track a head pose of the user.

In a 50th aspect, the method of aspect 48 or aspect 49, furthercomprising: identifying a head pose vector corresponding to the headpose of the user; and identifying a reference head pose vectorcorresponding to the reference head pose.

In a 51st aspect, the method of any of aspect 50, further comprising:determining an angular difference between the head pose vector and thereference head pose vector based at least in part on the comparisonbetween the head pose and the reference head pose, wherein the angulardifference corresponds to at least one of a difference in yaw, pitch, orroll of the head pose of the user with respect to the reference headpose.

In a 52nd aspect, the method of any one of aspects 48 to 51, whereinsaid determining the adjustment for the position of the virtual reticleis further based on a comparison of the angular difference to one ormore head pose thresholds.

In a 53rd aspect, the method of aspect 52, wherein the one or more headpose thresholds comprises at least one of a maximum head pose thresholdor a minimum head pose threshold.

In a 54th aspect, a method of adjusting a position of a movableindicator identifying a position of the user within a field of view ofthe user with respect to a display of a display system, the methodcomprising: identifying at least one of a max head pitch threshold, amin head pitch threshold, or a neutral head pitch threshold; identifyinga head pose vector corresponding to a head pose of a user; identifying areference head pose vector; calculating an angular difference betweenthe head pose vector and the reference head pose vector; calculating areticle adjustment based at least in part on the angular difference andat least one of the max head pitch threshold, the min head pitchthreshold, or the neutral head pitch threshold; and determining, basedat least in part on the calculated reticle adjustment, an adjustedreticle position.

In a 55th aspect, the method of aspect 54, further comprising obtaininghead pose data of the user of the display system from a head pose sensorconfigured to track a head pose of the user.

In a 56th aspect, the method of aspect 54 or aspect 55, wherein saiddetermining the angular difference is based at least in part on acomparison between the head pose vector and the reference head posevector, wherein the angular difference corresponds to at least one of adifference in yaw, pitch, or roll of the head pose of the user withrespect to the reference head pose.

In a 57th aspect, the method of any one of aspects 54 to 56, whereinsaid determining the adjusted reticle position is further based on acomparison of the angular difference to one or more head posethresholds.

In a 58th aspect, the method of aspect 57, wherein the one or more headpose thresholds comprises at least one of a maximum head pose thresholdor a minimum head pose threshold.

In a 59th aspect, a method of adjusting a position of a virtual reticleidentifying a position of a user within a field of view corresponding toa display of a display system, the method comprising: calculating anangular difference between a head pose vector and a reference head posevector, wherein the head pose vector corresponds to a head pose of auser of a display system, and wherein the reference head pose vectorcorresponds to a reference head pose; determining that the angulardifference does not satisfy a minimum head pitch threshold; anddetermining an adjusted reticle position based at least in part on anangle adjustment associated with the min head pitch threshold.

In a 60th aspect, the method of aspect 59, further comprising: obtaininghead pose data of the user of the display system from a head pose sensorconfigured to track a head pose of the user; identifying the head posevector corresponding to a head pose of the user based at least in parton the head pose data; and identifying the reference head pose vectorcorresponding to the reference head pose.

In a 61st aspect, the method of aspect 59 or aspect 60, wherein saidcalculating the angular difference is based at least in part on acomparison between the head pose vector and the reference head posevector, wherein the angular difference corresponds to at least one of adifference in yaw, pitch, or roll of the head pose of the user withrespect to the reference head pose.

In a 62nd aspect, a method of adjusting a position of a virtual reticleidentifying a position of a user within a field of view corresponding toa display of a display system, the method comprising: calculating anangular difference between a head pose vector and a reference head posevector, wherein the head pose vector corresponds to a head pose of auser of an display system, and wherein the reference head pose vectorcorresponds to a reference head pose; determining that the angulardifference satisfies a minimum head pitch threshold; determining thatthe angular difference does not satisfy a maximum head pitch threshold;and determining an adjusted reticle position based at least in part onan angle adjustment associated with the max head pitch threshold.

In a 63rd aspect, the method of aspect 62, further comprising: obtaininghead pose data of the user of the display system from a head pose sensorconfigured to track a head pose of the user; identifying the head posevector corresponding to a head pose of the user based at least in parton the head pose data; and identifying the reference head pose vectorcorresponding to the reference head pose.

In a 64th aspect, the method of aspect 62 or aspect 63, wherein saidcalculating the angular difference is based at least in part on acomparison between the head pose vector and the reference head posevector, wherein the angular difference corresponds to at least one of adifference in yaw, pitch, or roll of the head pose of the user withrespect to the reference head pose.

In a 65th aspect, a method of adjusting a position of a virtual reticleidentifying a position of a user within a field of view corresponding toa display of a display system, the method comprising: calculating anangular difference between a head pose vector and a reference head posevector, wherein the head pose vector corresponds to a head pose of auser of an display system, and wherein the reference head pose vectorcorresponds to a reference head pose; determining that the angulardifference satisfies a minimum head pitch threshold; determining thatthe angular difference does not satisfy a neutral head pitch threshold;and determining an adjusted reticle position based at least in part onan easing function.

In a 66th aspect, the method of aspect 65, further comprising: obtaininghead pose data of the user of the display system from a head pose sensorconfigured to track a head pose of the user; identifying the head posevector corresponding to a head pose of the user based at least in parton the head pose data; and identifying the reference head pose vectorcorresponding to the reference head pose.

In a 67th aspect, the method of aspect 65, wherein said calculating theangular difference is based at least in part on a comparison between thehead pose vector and the reference head pose vector, wherein the angulardifference corresponds to at least one of a difference in yaw, pitch, orroll of the head pose of the user with respect to the reference headpose.

In a 68th aspect, a method of adjusting a position of a virtual reticleidentifying a position of a user within a field of view corresponding toa display of a display system, the method comprising: calculating anangular difference between a head pose vector and a reference head posevector, wherein the head pose vector corresponds to a head pose of auser of a display system, and wherein the reference head pose vectorcorresponds to a reference head pose; determining that the angulardifference satisfies a neutral head pitch threshold; determining thatthe angular difference does not satisfy a maximum head pitch threshold;and determining an adjusted reticle position based at least in part onan easing function.

In a 69th aspect, the method of aspect 68, further comprising: obtaininghead pose data of the user of the display system from a head pose sensorconfigured to track a head pose of the user; identifying the head posevector corresponding to a head pose of the user based at least in parton the head pose data; and identifying the reference head pose vectorcorresponding to the reference head pose.

In a 70th aspect, the method of aspect 68 or aspect 69, wherein saidcalculating the angular difference is based at least in part on acomparison between the head pose vector and the reference head posevector, wherein the angular difference corresponds to at least one of adifference in yaw, pitch, or roll of the head pose of the user withrespect to the reference head pose. Any of the methods described in anyof the preceding aspects can be performed under control of a hardwareprocessor, for example, a hardware processor associated with ahead-mounted display system.

In a 71st aspect, a system comprising: a head pose sensor configured tomeasure head pose data of a user of the system; non-transitory memoryconfigured to store the head pose data corresponding to at least one ofan indication of a yaw, pitch, or roll of the head of the user; adisplay configured to be positioned in front of an eye of a user, thedisplay configured to project a virtual reticle toward the eye of theuser, wherein the virtual reticle comprises a movable indicatoridentifying a position of the user within a field of view; a hardwareprocessor in communication with the head pose sensor, the display, andthe non-transitory memory, the hardware processor programmed to: obtainthe head pose data of the user; identify a head pose vectorcorresponding to a head pose of the user based at least in part on thehead pose data; identify a reference head pose vector corresponding to areference head pose; determine an angular difference between the headpose vector and the reference head pose vector based at least in part ona comparison between the head pose of the user and the reference headpose, wherein the angular difference corresponds to at least one of adifference in yaw, pitch, or roll of the head pose of the user withrespect to the reference head pose; compare the determined angulardifference to one or more head pose thresholds, wherein the one or morehead pose thresholds comprises at least one of a maximum head posethreshold or a minimum head pose threshold; responsive to adetermination that the angular difference fails to satisfy the minimumhead pose threshold, determine an adjustment for a position of thevirtual reticle based at least in part on a first adjustment value;responsive to a determination that the angular difference satisfies themaximum head pose threshold, determine the adjustment for the positionof the virtual reticle based at least in part on a second adjustmentvalue; responsive to a determination that the angular differencesatisfies the minimum head pose threshold and fails to satisfy themaximum head pose threshold, determine the adjustment for the positionof the virtual reticle based at least in part on a third adjustmentvalue; and cause the position of the virtual reticle to be adjusted froma default reticle position of the field of view of the user based on thedetermined adjustment.

ADDITIONAL CONSIDERATIONS

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

APPENDIX A [Header(“Pitches limits relative to horizontal”)] publicfloat maxHeadPitch = 5; public float neutralHeadPitch = −20; publicfloat minHeadPitch = −45; [Header(“Angle offsets at the min/max headpitch”)] public float maxHeadPitchDelta = 8; public floatminHeadPitchDelta = −12; public Vector3 GetAcceleratedForwardVector( ) {// Pitch limits relative to horizontal float maxHeadPitch = 5; floatneutralHeadPitch = −20; float minHeadPitch = −45; // Angle offsets atthe min/max head pitch float maxHeadPitchDelta = 8; floatminHeadPitchDelta = −12; // Get the camera/headpose forward vectorVector3 forward = Headpose.Forward; // Get the forward vector projectedon to the horizontal plane Vector3 flat = forward; flat.y = 0;flat.Normalize ( ); // Calculate the absolute angle between the headposeforward vector and the flattened forward vector float pitch = Mathf.Acos(Vector3.Dot (forward, flat)) * Mathf.Rad2Deg; // If the forward vectoris pointing downwards, then the angle is negative if (forward.y < 0) {pitch = −pitch; } // pitchAdjustment is the angular deviation from theforward vector float pitchAdjustment = 0; if (pitch >= maxHeadPitch) {// If pitch > maxHeadPitch, set adjustment to maxHeadPitchDeltapitchAdjustment = maxHeadPitchDelta; } else if (pitch >=neutralHeadPitch) { // If pitch is between neutralHeadPitch andmaxHeadPitch, set adjustment to be an eased value between 0 andmaxHeadPitchDelta float t = (pitch − neutralHeadPitch) / (maxHeadPitch −neutralHeadPitch); t = EaseOutSine(t); pitchAdjustment =maxHeadPitchDelta * t; } else if (pitch >= minHeadPitch) { // If pitchis between neutralHeadPitch and minHeadPitch, set adjustment to be aneased value between 0 and minHeadPitchDelta float t = (pitch −minHeadPitch) / (neutralHeadPitch − minHeadPitch); t = EaseOutSine(1 −t); pitchAdjustment = minHeadPitchDelta * t; } else { // If pitch <minHeadPitch, set adjustment to minHeadPitchDelta pitchAdjustment =minHeadPitchDelta; } // If the pitch adjustment is not equal to zero if(pitchAdjustment != 0) { // Calculate the vector perpendicular to the upand forward vectors Vector3 right = Vector3.Cross (Vector3.up, flat); //Calculate the rotation adjustment Quaternion adjust =Quaternion.AngleAxis(pitchAdjustment, −right); // Apply the adjustmentto the forward vector to get a new adjusted forward vector forward =adjust * forward; } return forward; }  ©MAGIC LEAP, INC. All rightsreserved.

What is claimed is:
 1. A system comprising: a sensor configured toobtain an orientation of a head and/or eyes of a user of the system;non-transitory memory configured to store orientation data; a displayconfigured to be positioned in front of an eye of a user, and configuredto project a virtual reticle toward the eye of the user within a fieldof view (FOV) of the user, wherein the FOV changes as a head pose of theuser changes; a hardware processor in communication with the sensor, thedisplay, and the non-transitory memory, the hardware processorprogrammed to: recognize an orientation of the user's head and/or eyes;determine whether the orientation of the user's head and/or eyes areoutside of a range of acceptable head and/or eye poses; at least partlyin response determining that the orientation of the user's head and/oreyes are outside of the range of acceptable head and/or eye poses,reduce a degree to which the user needs to reorient the user's neck toalign the virtual reticle and a target object by: accelerating movementof the virtual reticle away from a default position and toward aposition in a direction of the user's head and/or eye movement.
 2. Thesystem of claim 1, wherein determining whether the orientation of theuser's head and/or eyes are outside of a range of acceptable head and/oreye poses further comprises determining whether the acceptable headand/or eye poses satisfies a first orientation threshold.
 3. The systemof claim 1, wherein while the orientation of the user's head and/or eyesare outside of a range of acceptable head and/or eye poses, a locationof the virtual reticle is based at least in part on an offset and posedata.
 4. The system of claim 1, wherein when the orientation of theuser's head is outside of a range of acceptable head poses, a locationof the virtual reticle is based at least in part on an easing functionand pose data.
 5. The system of claim 1, wherein while the orientationof the user's head and/or eyes are outside of a range of acceptable headand/or eye poses, the hardware processor is programmed to cause thedisplay to maintain the virtual reticle at a first location within theFOV.
 6. The system of claim 1, wherein the virtual reticle comprises amovable indicator identifying a position within in the FOV.
 7. Thesystem of claim 1, wherein the head pose is determined in an axial planeof the user, a coronal plane of the user, and a sagittal plane of theuser.
 8. A system comprising: a sensor configured to obtain anorientation of a head and/or eyes of a user of the system;non-transitory memory configured to store orientation data; a displayconfigured to be positioned in front of an eye of a user, and configuredto project a virtual reticle toward the eye of the user within a fieldof view (FOV) of the user, wherein the FOV changes as a head pose of theuser changes; a hardware processor in communication with the sensor, thedisplay, and the non-transitory memory, the hardware processorprogrammed to: recognize an orientation of the user's head and/or eyes;determine whether the orientation of the user's head and/or eyes areoutside of a range of acceptable head and/or eye poses; at least partlyin response determining that the orientation of the user's head and/oreyes are outside of the range of acceptable head and/or eye poses,reduce a degree to which the user needs to reorient the user's neck toalign the virtual reticle and a target object by: causing the virtualreticle to move away from a default position and toward a position in adirection of the user's head and/or eye movement at least partly inresponse determining that the orientation of the user's head and/or eyesare within the range of acceptable head and/or eye poses causing thedisplay to render a virtual reticle at a fixed location within the FOV.9. The system of claim 8, wherein determining whether the orientation ofthe user's head and/or eves are outside of a range of acceptable headand/or eye poses further comprises determining whether the acceptablehead and/or eye poses satisfies a first orientation threshold.
 10. Thesystem of claim 8, wherein while the orientation of the user's headand/or eyes are outside of a range of acceptable head and/or eye poses,a location of the virtual reticle is based at least in part on an offsetand pose data.
 11. The system of claim 8, wherein when the orientationof the user's head is outside of a range of acceptable head poses, alocation of the virtual reticle is based at least in part on an easingfunction and pose data.
 12. The system of claim 8, wherein while theorientation of the user's head and/or eyes are outside of a range ofacceptable head and/or eye poses, the hardware processor is programmedto cause the display to maintain the virtual reticle at a first locationwithin the FOV.
 13. The system of claim 8, wherein the virtual reticlecomprises a movable indicator identifying a position within in the FOV.14. The system of claim 8, wherein the head pose is determined in anaxial plane of the user, a coronal plane of the user, and a sagittalplane of the user.
 15. A method of adjusting a position of a virtualreticle identifying a position within a field of view (FOV)corresponding to a display of a display system, the method comprising:recognizing an orientation of a user's head and/or eyes based at leaston data from one or more sensors; determining whether the orientation ofthe user's head and/or eyes are outside of a range of acceptable headand/or eye poses; at least partly in response determining that theorientation of the user's head and/or eyes are outside of the range ofacceptable head and/or eye poses, reducing a degree to which the userneeds to reorient the user's neck to align the virtual reticle and atarget object by: accelerating movement of the virtual reticle away froma default position and toward a position in a direction of the user'shead and/or eye movement; at least partly in response determining, thatthe orientation of the user's head and/or eyes are within the range ofacceptable head and/or eye poses causing the display to render a virtualreticle at a fixed location within the FOV.
 16. The method of claim 15,wherein determining whether the orientation of the user's head and/oreyes are outside of a range of acceptable head and/or eye poses furthercomprises determining whether the acceptable head and/or eye posessatisfies a first orientation threshold.
 17. The method of claim 15,wherein while the orientation of the uses head and/or eyes are outsideof a range of acceptable head and/or eye poses, a location of thevirtual reticle is based at least in part on an offset and pose data.18. The method of claim 15, wherein when the orientation of the user'shead is outside of a range of acceptable head poses, a location of thevirtual reticle is based at least in part on an easing function and posedata.
 19. The method of claim 15, wherein while the orientation of theuser's head and/or eyes are outside of a range of acceptable head and/oreye poses, the method further comprises causing the display to maintainthe virtual reticle at a first location within the FOV.
 20. The methodof claim 15, wherein the virtual reticle comprises a movable indicatoridentifying a position within in the FOV.
 21. The method of claim 15,wherein the head pose is determined in an axial plane of the user, acoronal plane of the user, and a sagittal plane of the user.